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I.  INTRODUCTION 


Cost  growth  in  Department  of  Defense  (DoD)  aequisition  programs  has  been  of 
great  concern  to  virtually  every  administration  and  Congress  since  the  1970s. 
Periodically,  new  procedures  are  adopted  to  control  it,  new  horror  stories  emerge,  and 
new  studies  are  undertaken  to  reveal  its  causes.  The  latest  surge  of  concern  began  in 
2008,  with  the  release  of  a  Government  Accountability  Office  (GAO)  report  that  DoD’s 
Major  Defense  Acquisition  Programs  (MDAPs)  had  grown  in  cost  by  an  estimated  $296 
billion.  1  With  a  new  administration  on  the  horizon,  the  Director  for  Acquisition 
Resources  and  Analysis,  in  the  office  of  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  (OUSD(AT&L)),  turned  to  IDA  for  an  in-depth  examination  of 
the  causes  of  DoD  acquisition  cost  growth. 

The  report  is  organized  into  three  volumes.  Volume  I  is  a  self-contained  executive 
summary;  volume  II  (this  volume)  describes  the  methodology  employed  in  the  study, 
summarizes  the  formal  acquisition  process  and  available  data  sources,  and  presents  the 
findings,  conclusions,  and  recommendations  which  have  emerged  to  the  study  team 
through  examination  of  the  1 1  programs.  They  are  supported  with  examples  from  the  1 1 
programs.  Volume  III  consists  of  13  appendices,  one  for  each  of  the  programs  examined, 
with  a  narrative  of  the  program’s  history  and  discussion  of  how  the  insights  supporting 
the  findings  and  conclusions  emerged,  a  list  of  references,  and  a  list  of  acronyms  and 
abbreviations. 

In  this  introductory  chapter,  we  describe  how  this  study  differs  from  and  builds 
upon  its  predecessors,  provide  an  overview  of  the  Defense  acquisition  system  and  the 
historical  forces  that  shaped  it,  and  briefly  discuss  the  strengths  and  weaknesses  of  the 
data  that  serve  as  the  basis  for  our  analysis. 


^  Government  Accountability  Office,  “Fundamental  Changes  are  Needed  to  Improve  Weapon  Program 
Outcomes,”  September,  2008,  GAO-08-1159T.  While  the  accuracy  of  this  figure  is  not  disputed,  the 
interpretation  of  it  has  been.  A  March  31,  2009,  memorandum  from  the  USD(AT&L)  to  the  Secretary 
of  Defense  says:  “...I  would  suggest  that  the  number  is  misleading,  out-of-date,  and  largely  irrelevant 
to  the  current  management  of  DoD  programs.” 


1 


A.  WHY  IS  THIS  STUDY  DIFFERENT? 


This  report  identifies  the  primary  causes  of  major  cost  growth  through  case 
studies  of  1 1  MDAPs  and  offers  recommendations  for  better  controlling  cost  growth  in 
future  programs.  It  identifies,  with  the  full  benefit  of  hindsight,  particular  decisions  and 
mistakes  that  resulted  in  cost  growth,  not  with  the  intent  of  blaming  individuals,  but  to 
illuminate  problems  in  the  DoD  acquisition  process  that  point  the  way  to  the 
recommendations  for  improvements.  In  this  context  it  is  important  that  the  reader  keep  in 
mind  that  successful  Defense  acquisitions  depend  on  the  efforts  of  many  participants  in 
the  process.  The  Defense  Acquisition  Executive  (DAE),  to  be  sure,  has  primary 
responsibility  for  the  key  decisions  on  the  major  programs  and  for  the  overall  functioning 
of  the  acquisition  system,  but  the  quality  and  successful  implementation  of  his  decisions 
depend  on  many  others — the  supporting  staffs  in  the  Office  of  the  Secretary  of  Defense 
(OSD),  the  Joint  Staff,  the  Services,  including  the  Service  Acquisition  Executives 
(SAEs),  government  program  managers  and  contracting  officials,  and  defense 
contractors.  This  large  ensemble  of  participants  must  work  well  together  as  a  team  to 
provide  the  information  and  analyses  that  support  good  decisions  and  carry  them  out  once 
they  are  made. 

Moreover,  decisions  about  MDAPs  touch  on  the  core  interests  of  important 
stakeholders  outside  the  formal  acquisition  process  both  inside  and  outside  DoD.  The 
DAE  must  exercise  a  keen  understanding  of  those  forces,  including  the  transcendental 
goals  of  the  administration  of  which  he  is  a  key  member.  This  report  is  intended,  not  to 
indict  the  past,  but  to  help  future  decision-makers  avoid  the  pitfalls  it  identifies. 

This  study  sought  a  deeper  understanding  of  the  decisions  and  mistakes  that 
contribute  to  cost  growth  through  an  in-depth  examination  selected  MDAPs.  Working 
with  the  sponsor,  IDA  selected  11  important  programs  that  had  entered  full-scale 
engineering  development  between  1995  and  2006  and  subsequently  experienced  cost 
growth  that  breached  or  nearly  breached  the  thresholds  established  by  the  Nunn- 
McCurdy  Act.2  The  study  team  assembled  histories  of  the  programs  by  examining 
official  records  and  through  interviews  with  former  senior  acquisition  officials  and  their 
staffs,  cost  estimators  and  analysts  in  the  Office  of  the  Director  Program  Analysis  and 
Evaluation  (PA&E — recently  renamed  Cost  Assessment  and  Program  Evaluation 


2  The  Nunn-McCurdy  Act  requires  the  Department  to  make  a  new  determination  of  need  and  funding 
adequacy  for  programs  whose  projected  costs  rise  25%  above  their  baseline  estimates. 
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(CAPE)),  and  personnel  in  the  military  departments.  This  report  presents  the  findings  of 
those  analyses,  general  eonelusions  regarding  the  sources  and  contributing  factors  of  cost 
growth,  and  recommendations  on  how  to  constrain  acquisition  cost  growth  in  the  future. 

We  relied  heavily  on  Selected  Acquisition  Reports,^  Defense  Acquisition 
Executive  System  (DAES)  reports,  and  Acquisition  Decision  Memoranda  (ADMs). 
Additional  sources  included  GAO  reports.  Congressional  Research  Service  and 
Congressional  Budget  Office  studies,  documentation  from  program  management  offices 
(PMOs)  obtained  from  OUSD(AT&E)  staff,  and  miscellaneous  sources  on  individual 
programs. 

Over  100  people  were  interviewed.  The  general  ground  rule  was  “non¬ 
attribution.”  The  range  of  interviewees  spanned  from  senior  defense  decision-makers 
(e.g.  former  USD  (AT&L)s  and  principal  deputies),  through  mid-level  managers,  to  staff 
action  officers.  Most  interviews  were  with  people  in  OUSD(AT&E)  and  the  Cost 
Analysis  Improvement  Group  (CAIG)^  but  interviews  were  also  conducted  with 
personnel  in  other  OSD  offices,  including  offices  under  the  PA&E  Director’s 
supervision.  A  number  of  key  people  in  the  Military  Services  were  included,  but  contact 
with  current  PMOs  was  deliberately  limited.  ^  While  respecting  the  “non- attribution”  rule, 
we  will  attempt,  in  so  far  as  possible  in  this  document,  to  indicate  when  a  finding  or 
conclusion  is  based  on  the  informed  opinions  of  interviewees.  In  general,  we  sought 
corroboration  from  at  least  two  independent  sources  before  drawing  conclusions  based  on 
the  opinions  of  interviewees. 

In  addition,  some  of  the  members  of  the  IDA  study  team  were  familiar  with  some 
of  the  programs  based  on  their  past  experience  in  government  and  industry. 


3  Because  considerable  care  goes  into  preparing  the  SAR  reports  to  Congress,  and  no  other  systematic 
and  accessible  tracking  system  exists,  the  SAR  data  base  is,  in  almost  all  cases,  the  singular  and  most 
definitive  source  of  information  for  DoD  cost  growth  studies. 

^  The  Weapon  Systems  Acquisition  Reform  Act  of  2009  directed  a  reorganization  of  the  CAIG,  which  is 
to  be  renamed  as  Director  of  Cost  Assessment  and  Program  Evaluation  (with  a  Deputy  Director  for 
Program  Evaluation).  This  report,  however,  was  conducted  for  the  most  part  before  the  law  was 
implemented,  so  the  terminology  in  effect  at  the  time  will  be  employed. 

^  The  JSF  Program  Office  was  quite  helpful,  however. 
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B.  THE  SELECTED  PROGRAMS 


All  11  programs,  shown  in  Table  1  with  their  current  status,  entered  full-scale 
development  at  least  four  years  ago.® 

Table  1.  Acquisition  Programs  Examined 


System 

Date  of  MS- 
II/B  Approval 

Current  Status 

Armed  Reconnaissance  Helicopter 
(ARH) 

Jul-05 

Cancelled 

Expeditionary  Fighting  Vehicle  (EFV) 

Dec-00 

Pre-MS  C 

Future  Combat  System  (FCS) 

May-03 

Pre-MS  C 
(Restructured) 

Global  Hawk 

Feb-OI 

TRIP* 

Joint  Air-to- Surface  Standoff  Missile 

Nov-98 

TRIP 

(JASSM) 

Joint  Strike  Fighter  (JSF) 

Oct-OI 

TRIP  -4 

Joint  Tactical  Radio  System  (JTRS)- 
Ground  Mobile  Radio  (GMR) 

Jun-02 

Pre-MS  C 

Fittoral  Combat  Ship  (FCS) 

May-04** 

Pre-MS  B 

Amphibious  Transport  Dock  (FPD-17) 

Jun-96 

In  production 

Space-Based  Infrared  Satellite-High 
(SBIRS) 

Oct-96 

Pre-MS  C 

Warfighter  Information  Network- 
Tactical  (WIN-T) 

Jul-03 

Pre-MS  C*** 

*  Low-rate  initial  production  **  Milestone  A  ***  Restructured.  Original  WIN-T  program  now 
divided  into  two  programs—  WIN-T  Increment  2  and  WIN-T  Increment-3 


®  We  use  this  term  generieally,  sinee  the  name  has  ehanged  several  times — from  Engineering  and 
Manufaeturing  Development  (EMD)  to  System  Development  and  Demonstration  (SDD)  and  baek 
again.  We  will  use  either  EMD  or  SDD,  whiehever  term  was  in  use  at  the  time.  Likewise,  when 
diseussing  speeifie  systems  we  will  refer  to  the  nomenelature  in  use  at  the  time. 
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C.  THE  COST  METRIC 


Normally  cost  growth  is  measured  from  the  point  when  the  program  is  approved 
for  entry  into  full-scale  development.  The  decision  point  is  now  known  as  Milestone  B 
(MS  B),  and  is  generally  the  point  at  which  the  first  full  cost  reports  are  provided  to  the 
Congress.  This  is  the  point  at  which  the  program  beeomes  what  is  known  as  a  “program 
of  reeord”  within  the  aequisition  process.  In  one  program,  littoral  combat  ship  (LCS),  MS 
B  has  not  yet  occurred,  and  cost  growth  is  measured  from  Milestone  A  (MS  A),  as  it  is 
with  most  other  ship  programs.^  For  eertain  other  programs  (e.g..  Joint  Strike  Fighter 
(JSF))  pre-milestone  II  or  B  eost  estimates  are  of  interest. 

We  used  Program  Aequisition  Unit  Cost  (PAUC)  as  the  primary  metric  for  cost 
growth.  PAUC  is  the  sum  of  all  investment  eosts  (RDT&E,  Procurement,  and  Military 
Construction)  in  constant-year  dollars  divided  by  the  aequisition  quantity  planned  at  the 
time  of  the  estimate.  The  PAUC  is  the  single  most  representative  measure  for  eomparing 
cost  growth  in  programs  with  highly  diverse,  particularly  with  regard  to  the  relative 
magnitudes  of  R&D  and  production  costs.  This  diversity  reflects  the  broad  span  from 
programs  that  eost  a  lot  to  develop,  but  entail  the  production  of  only  a  few  “end  items” 
sueh  as  satellites  and  large  new  software  programs  that  run  on  existing  or  low-cost 
equipment,  to  programs  entailing  thousands  of  new  end  items  after  only  a  limited  amount 
of  development,  sueh  as  new  models  of  expendable  weapons. 

Figure  1  depicts  the  cost  growth  to  date  in  the  1 1  programs. 


^  Ship  programs  are  formally  started  and  traeked  from  MS  A  primarily  beeause  that  is  the  point  of 
eommitment  to  aetually  build  and  field  the  first  ship. 
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Figure  1.  Cost  Growth  in  Seiected  Acquisition  Programs 


Except  for  ECS,  the  ehart  displays  PAUC  growth  from  eaeh  program’s  initial 
formal  estimate  to  its  most  reeent  estimate.  Eor  those  programs  that  have  been  re- 
baselined,^  the  eost  inerements  are  shown  for  eaeh  revision.  Where  the  eurrent  estimate 
exeeeds  the  most  reeent  baseline  estimate,  the  differenee  is  shown  as  an  additional 
inerement.  Programs  are  re -baselined  to  aeknowledge  signifieant  ehanges  in  seope,  other 
irreversible  eost  growth  and/or  schedule  slippage.  A  program  may  hold  to  its  MS  B 
baseline  for  an  extended  period  either  beeause  it  has  not  eneountered  signifieant  eost  or 
sehedule  problems  or  beeause  the  problems  go  unreeognized  or  are  ignored.  Examples  of 
the  latter  eases  will  be  diseussed  in  Chapter  II. 

The  average  PAUC  growth  for  these  1 1  programs  was  90%,  ranging  from  a  low 
of  19%  for  Joint  Taetieal  Radio  System  (JTRS)  to  a  high  of  254%  for  SBIRS.  The 


The  “baseline”  is  the  time -phased  spending  plan  and  associated  events  that  are  approved  by  the 
Milestone  Decision  Authority  (MDA)  at  program  inception  -  usually  MS  B. 
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aggregate  total  program  cost  growth  is  $99  billion  (in  constant  FY2005  dollars).  It  is 
extremely  important  to  note  that  this  dollar  figure  is  highly  theoretical — it  represents  the 
estimated  total  investment  costs  over  the  entire  life  of  the  acquisition  phase  of  the 
programs  (including  both  future  and  sunk  costs),  if  the  current  cost  estimates  prove 
accurate  and  currently  projected  procurement  quantities  are  unchanged. 

The  5000  series  directives  lay  out  a  normal  sequence  of  acquisition  program 
phases,  demarcated  by  major  reviews  (called  Milestones)  and  other  decision  points. 
These  are  shown  in  a  schematic  outline  in  Figure  2,  as  they  have  evolved  through  various 
iterations  of  the  5000  series  over  the  period  reviewed  in  this  study. 

In  all  of  these  diagrams,  the  triangles  indicate  formal  milestone  reviews  by  the 
Defense  Acquisition  Board  (DAB).  The  DAB  is  chaired  by  the  Under  Secretary  of 
Defense  (AT&L),  acting  in  his  capacity  as  the  DAE.  The  vice  Chairman  of  the  Joint 
Chiefs  of  Staff  (VCJCS)  serves  as  vice  chairman,  while  other  principal  members  include 
the  principal  direct  subordinates  of  the  DAE,  and  the  Director,  PA&E  (now  CAPE),  the 
Director,  OT&E,  the  DoD  Comptroller,  representative  of  other  concerned  OSD  staff 
elements,  and  the  Secretaries  of  the  Military  Departments. 

D,  OVERVIEW  OF  THE  DOD  ACQUISITION  PROCESS 

This  section  briefly  reviews  the  features  of  the  DoD  acquisition  system  that  are 
directly  relevant  to  the  content  of  this  report. 

In  addition  to  applicable  law,  DoD’s  acquisition  process  is  governed  by  a  linked 
set  of  directives,  known  collectively  as  the  “5000  series.”  They  were  first  approved  in 
their  current  format  in  July  1971  and  have  evolved  over  the  course  of  12  generations. ^ 
The  versions  adopted  in  September  1987  departed  substantially  from  previous  versions’ 
consonance  with  the  recommendations  of  the  1986  Packard  Commission  repoifio  and  the 
Goldwater-Nichols  Department  of  Defense  Reorganization  Act  of  1986.”  The  principal 
recommendations  of  the  Packard  report  can  be  succinctly  characterized  as  “Do  it  right  the 


^  The  current  DoD  Directive  5000.01,  dated  November  20,  2007,  is  an  updated  version  of  an  earlier 
version  issued  in  May,  2003,  with  minor  and  administrative  changes.  The  current  DoD  Instruction 
5000.02  was  issued  December  8,  2008.  The  first  11  generations  of  the  5000  series  are  documented  in 
“The  Evolution  of  DoD  Directive  5000.1  Acquisition  Management  Policy,  1971  to  2003,”  Acquisition 
History  Project  Working  Paper  #3,  http://www.historv.armv.mil/acquisition/research/working3.html. 

A  Quest  for  Excellence:  Final  Report  to  the  President  by  the  President’s  Blue  Ribbon  Commission  on 
Defense  Management,  June  1986. 

1 1  http://www.ndu.edu/hbrary/goldnich/goldnich.html. 
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first  time”  and  “Do  it  within  clear,  short  lines  of  decision  authority.”  This  basic  approach 
was  repeated  with  some  variations  in  the  versions  issued  in  1991  and  1996.  Additional 
changes  were  made  in  2000,  and  again  in  2003.  Since  all  of  the  decisions  that  are  relevant 
to  this  study  were  taken  under  the  1991,  1996,  2000,  or  2003  versions,  we  briefly  review 
their  main  provisions  as  they  apply  to  the  programs  we  studied. 

This  report  is  concerned  specifically  with  MDAPs,  a  category  defined  by  law  to 
include  acquisition  programs  that  are  estimated  to  require  an  eventual  total  expenditure  in 
Fiscal  Year  (FY)  2000  constant  dollars  of  more  than 

•  $365  million  in  Research,  Development,  Test  and  Evaluation  (RDT&E)  or 

•  $2.19  billion  in  procurement. 

All  of  the  programs  we  studied  exceeded  the  cost  thresholds  and  have  been 
designated  as  MDAPs.  All  MDAPs  fall  under  Acquisition  Category  I  (AC AT  I).i3 

1.  Acquisition  Program  Phases  and  Milestones 

The  5000  series  directives  lay  out  a  “normal”  sequence  of  acquisition  program 
phases,  demarcated  by  major  reviews  (called  Milestones)  and  other  decision  points. 
These  are  shown  in  a  schematic  outline  in  Eigure  2,  as  they  have  evolved  through  various 
iterations  of  the  5000  series  over  the  period  reviewed  in  this  study. 

In  all  of  these  diagrams  the  triangles  indicate  formal  milestone  reviews  by  the 
Defense  Acquisition  Board  (DAB).  The  DAB  is  chaired  by  the  Under  Secretary  of 
Defense  (AT&E),  acting  in  his  capacity  as  the  DAE.  The  Vice  Chairman  of  the  Joint 
Chiefs  of  Staff  (VCJCS)  serves  as  vice  chairman,  while  other  principal  members  include 
the  principal  direct  subordinates  of  the  DAE,  and  the  Director,  PA&E  (now  CAPE),  the 
Director,  OT&E,  the  DoD  Comptroller,  representatives  of  other  concerned  OSD  staff 
elements,  and  the  Secretaries  of  the  Military  Departments. 


The  relevant  law  is  codified  as  10  USC  §2430.  The  base  year  is  changed  by  DoD  action  from  time  to 
time,  with  amounts  adjusted  by  the  officially  approved  deflators,  as  provided  by  the  law. 

12  Acquisition  categories  are  established  by  the  5000  series  directives  to  facilitate  decentralized  decision 
making  and  execution  and  compliance  with  statutorily  imposed  requirements.  The  categories  are 
determined  largely  by  the  cost  level  of  the  program,  and  determine  the  level  of  review,  decision 
authority,  and  applicable  procedures.  There  are  currently  four  top-level  categories  with  a  variety  of 
sub-categories  and  sub-sub-categories.  Generally  only  ACAT  ID  is  of  concern  in  this  study,  although 
JASSM  reverted  to  ACAT  IC  for  several  years. 
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Figure  2.  Acquisition  Process  as  Defined  in  DoD  5000  Series  Documents,  1991-2008 
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Although  it  has  had  no  effect  to  date  on  any  of  the  programs  reviewed  here,  we 
will  refer  later  to  the  version  of  the  acquisition  process  laid  out  in  the  revision  published 
in  December,  2008,  which  is  depicted  in  Error!  Reference  source  not  found.. 


Decision  Point  /\=  Miiestone  Review 


if\=  Decision  Point  if  PDR  is  not 
V  conducted  before  Miiestone  B 


Figure  3.  Acquisition  Process  as  Revised  by  DoDI  5000.02,  December  2008 

2,  Management  Responsibilities  and  Processes 

Each  MDAP  has  a  program  manager  (PM)  who  normally  reports  through  a 
program  executive  officer  (PEO)  to  a  Service  or  Component  acquisition  executive  (SAE 
or  CAE)  or  Component  head  who,  in  turn,  is  responsible  to  the  DAE.i^  The  SAEs  are 
assistant  secretaries  of  the  Military  Departments.  This  structure  resulted  from  the  Packard 
Commission  recommendation  that  the  department  strengthen  Defense  acquisition 
decision-making. 

Programs  must  receive  approval  to  proceed  at  each  milestone  from  the  Milestone 
Decision  Authority  (MDA).  Eor  ACAT  I  programs  (i.e.,  MDAPs)  the  MDA  is  normally 
the  DAE,  in  which  case  the  program  is  ACAT  ID.  If  he  sees  fit,  the  DAE  may  delegate 
milestone  decision  authority  to  a  SAE/CAE,  in  which  case  it  becomes  an  ACAT  IC 
program.  DAEs  typically  delegate  milestone  authority  for  major  programs  when  the 
residual  risk  is  assessed  as  minimal — for  example,  for  follow-on  production  decisions. 

A  program  becomes  an  acquisition  “Program  of  Record”  at  the  points  shown  in 
Eigures  2  and  3— at  Milestone  II  until  1996,  at  Milestone  I  from  1996  until  2000,  and  at 
Milestone  B  since  2000.  Eor  many  programs,  formal  initiation  came  after  several  years  of 
effort  costing  hundreds  of  millions  of  dollars  in  system  concept  and  technology 


This  management  strueture  was  established  by  the  Goldwater-Niehols  Aet. 
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development.  In  such  cases,  the  developed  concept  usually  has  so  much  momentum  (and 
support  from  vested  interests)  that  it  becomes  difficult  to  cancel  or  redirect. 

The  5000  series  directives  specify  criteria  that  govern  whether  and  when  a 
program  becomes  ready  for  Milestone  B  review.  The  criteria  have  varied  in  detail  over 
the  years,  but  their  most  important  (and  durable)  provisions  require: 

•  A  Component  recommendation  for  a  new  program  start,  approved  by  the  Joint 
Requirements  Oversight  Council  (JROC)  (see  additional  discussion  below); 

•  Demonstrated  readiness  of  the  key  component  technologies  of  the  proposed 
system  to  support  full-scale/engineering  and  manufacturing  development;  and 

•  That  the  sponsoring  Component  demonstrate  firm  plans  for  funding  the 
program  within  the  Component’s  fiscal  resources.!^ 

In  addition  to  these,  the  Acquisition  Decision  Memoranda  (ADMs)  for  Milestone 
A  (MS  A)  approvals  (if  held)  have  usually  specified  exit  criteria  that  the  program  must 
meet  before  being  considered  for  MS  B  approval. 

3.  Requirements,  JROC,  and  JCIDS 

Systems  developed  through  the  DoD  acquisition  process  are  intended  to  meet 
military  needs,  or  “requirements.”  For  the  vast  majority  of  MDAPs,  these  requirements 
are  developed  by  the  Military  Services  and  reviewed  by  the  JROC,  which  is  chaired  by 
the  Vice  Chairman  of  the  Joint  Staff.  The  Joint  Capabilities  Information  and 
Development  System  (JCIDS)  process  was  implemented  to  inform  the  process  of  JROC 
review.  After  its  review,  the  JROC  makes  a  recommendation,  which  is  normally 
forwarded  by  the  Chairman,  Joint  Chiefs  of  Staff,  to  the  DAE.  This  requirements 
identification  process,  which  was  developed  in  response  to  the  Goldwater-Nichols  Act, 
was  intended  to  reduce  the  dominance  of  the  Military  Services  in  determining  what  the 
Department  should  acquire  and  increase  the  emphasis  on  acquiring  equipment  to  support 
joint  operations.  The  process  has  been  revised  several  times  in  efforts  to  better  achieve 
those  ends.i^ 


The  mechanisms  for  doing  that  will  be  discussed  later  in  this  chapter  and  the  next. 

See  Government  Accountability  Office,  “DoD’s  Requirements  Determination  Process  Has  Not  Been 
Effective  in  Prioritizing  Joint  Capabilities,”  GAO-08-1060,  September  2008;  interviews  with  former 
high  officers  and  officials  wifh  direcf  knowledge. 
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Although  Section  105  of  the  Weapon  Systems  Acquisition  Reform  Act  of  2009 
(WSARA  2009)1^  calls  for  stronger  input  to  the  JROC  from  the  unified  combatant 
commanders  (CoComs),  the  CoComs  are  focused  on  operational  near-term  and  often 
geographic-area-specific  needs,  so  their  inputs  regarding  longer-range  acquisition 
programs  are  limited. 

Although  the  Secretary  of  Defense  clearly  has  the  authority  to  adopt  requirements 
other  than  those  the  Chairman  recommends  and  could  delegate  that  authority  to  the  DAE, 
that  authority  is  seldom  exercised  and  has  not  been  so  delegated. 

4,  Analysis  of  Alternatives 

The  Analysis  of  Alternatives  (AoAs)  is  the  process  of  justifying  that  a  proposed 
specific  new  acquisition  program  is  the  best  solution  for  the  need  that  has  been  identified. 
Strongly  related  to  acquisition  requirements,  AoAs,  according  to  DoDI  5000.02,  must  be 
completed  before  milestone  A,  B  and  C  reviews  (for  the  later  milestones  an  update  is 
required  if  a  previous  AoA  was  performed).  The  AoA  is  intended  to  evaluate  the 
proposed  program  in  comparison  to  alternatives,  such  as  upgrades  of  existing  systems, 
purchase  of  foreign  or  commercial  systems,  etc.  The  alternatives  considered  should  be 
selected  from  a  broad  set  for  early  AoAs  and  “necked  down”  for  later  ones.  For  example, 
for  a  mission  for  which  a  helicopter  has  been  proposed,  an  AoA  before  MS  A,  might 
consider  an  unmanned  air  vehicle,  whereas  before  MS  B,  only  helicopter  alternatives 
would  likely  be  considered  if  the  prior  AoA  had  convincingly  ruled  out  unmanned 
alternatives. 

A  recent  GAO  report  evaluated  a  number  of  AoAs,  including  those  for  ECS  and 
ARH,  and  found  them  wanting,  as  will  be  discussed  in  Chapter  II  and  the  appendixes.  We 
did  not,  however,  systematically  review  AoAs  for  this  study  —only  when  they  came  up  in 
the  context  of  cost  growth. 

5.  Technology  Readiness  and  Assessment 

It  is  DoD  policy  that  no  system  can  enter  full-scale  development  unless  the 
technologies  essential  to  its  military  effectiveness  are  mature  enough  to  support  the 
timely  completion  of  development  and  system  integration  at  an  acceptable  level  of  risk. 
Since  the  1960s,  DoD  acquisition  policy  has  emphasized  the  need  for  early  technology 


17  Public  Law  1 11-23,  May  22,  2009. 
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development  and  evaluation  to  reduce  risks.  As  indicated  in  Figures  2  and  3,  a  formal 
technology  development  phase  has  long  been  the  norm  for  MDAPs.  By  the  late  1990s  the 
need  for  a  formal  technology  readiness  assessment  (TRA)  process  had  become  evident. 
The  requirement  for  assessment  of  technology  readiness  became  institutionalized  in  the 
October  2000  revision  of  the  5000  series, 

Today  DoD  has  a  formal  system  for  grading  technology  readiness,  expressed  in 
terms  of  Technology  Readiness  Levels  (TRLs)  from  TRL  1  (basic  principles  observed 
and  reported)  through  TRL  9  (actual  system  proven  through  successful  mission 
operations).  19  Over  time,  particularly  since  2000,  the  governing  TRL  to  be  achieved 
before  entry  into  full-scale  engineering  development  has  become  more  definitive,  but  the 
actual  levels  of  technology  readiness  at  Milestone  B  have  varied  substantially.  The  FY 
2006  Defense  authorization  bill  directed  that  TRL  6  be  required  unless  a  national  security 
exception  is  granted  by  the  DAE.  The  WSARA  further  definitized  in  law  the  need  to 
assess  technology  readiness. 

The  process  for  assessing  the  soundness  of  the  overall  system  concept,  design, 
and  integration  plans  is  far  less  developed  than  that  for  technology  readiness,  and  no 
formal  mechanism  to  ensure  that  such  plans  are  well-developed  before  approval  to  enter 
full-scale  development  currently  exists. 

6,  Cost  Estimating  and  the  CAIG 

The  craft  of  cost  estimation  plays  a  central  role  in  program  planning  in  general, 
and  acquisition  program  planning  in  particular.  Until  the  late  1960s,  the  DoD  process  for 
estimating  the  cost  of  engineering  and  manufacturing  projects  began  with  analogies  to 
earlier  projects,  scaled  as  necessary  to  account  for  increases  in  size  and  complexity.  If  the 
rough  estimates  seemed  reasonable,  and  the  project  was  judged  to  be  a  cost  effective  way 
to  meet  a  real  need,  work  proceeded  on  the  design  and  planning  until  it  was  possible  to 
list  all  of  the  required  parts  and  materials  together  with  the  labor  needed  for  manufacture 
and  assembly.  This  provided  the  basis  for  a  detailed  “build-up”  or  “bottom-up”  estimate 


DoD  5000. 2-R,  “Mandatory  Procedures  for  Major  Defense  Acquisition  Programs  (MDAPs)  and  Major 
Automated  Information  System  (MAIS)  Acquisition  Programs,”  10  June  2001,  section  C7.5. 

19  Defined  in  “Defense  Acquisition  Guidebook,”  <https://acc.dau.mil/dag>.  Chapter  10,  Table  10.5.2.T1, 
accessed  13  July  2009.  Responsibility  is  assigned  to  the  Director,  Defense  Research  and  Engineering, 
OUSD(AT&L). 
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of  production  costs.  For  a  number  of  reasons,  this  approach  did  not  work  as  well  as 
expected. 

For  eomplex  systems,  sueh  as  aircraft  and  other  weapon  systems,  the  build-up 
approaeh  requires  a  level  of  knowledge  that  rarely  exists  in  the  early  stages  of  a  system’s 
detailed  design.  Furthermore,  early  experience  with  MDAPs  showed  that  build-up 
estimates  often  were  not  reliable  indicators  of  eventual  actual  costs.  A  refined  version  of 
eosting  by  analogy  was  found  to  be  superior.  This  involves  a  eombination  of  engineering 
design  reviews  and  statistical  analysis  of  cost  experience  in  other  projects,  known  as 
parametric  cost  estimation.  Parametric  cost  estimating  of  likely  production  costs  is  most 
reliable  when  it  can  draw  on  the  widest  possible  base  of  aeeurate  reeords  of  actual  project 
costs.  Parametric  estimates  for  MS  B  reviews  are  made  at  the  sub-system  level,  or  even  at 
the  level  of  individual  components,  and  then  rolled  up. 

Vocal  Congressional  dissatisfaction  with  reported  eost  growth  prompted  then- 
Seeretary  Laird  to  establish  the  CAIG  in  1972.  In  1983,  eontinuing  Congressional 
concerns  about  aequisition  eost  estimation  resulted  in  a  requirement  in  statute^o  for  the 
development  of  “Independent  Cost  Estimates”  (ICEs)  by  an  entity  independent  of  the 
DoD  Components  responsible  for  developing  the  systems.  The  Seeretary  of  Defense 
assigned  that  responsibility  to  the  CAIG,  but  initially  the  CAIG  was  not  adequately 
staffed  to  develop  independent  eost  estimates  for  all  of  the  programs  the  Department  was 
initiating.  The  CAIG  dealt  with  this  problem  by  developing  full-blown  ICEs  for  only  the 
most  important  programs,  and  reviewing  and  correeting  Service-developed  estimates  for 
other  programs — a  course  of  action  not  entirely  responsive  to  the  statute.^i  A  1990  DoD 
Inspector  General  review,  undertaken  in  response  to  eontinuing  eoneerns  about  aecuraey 
of  aequisition  program  costing,  resulted  in  a  clear  affirmation  of  the  CAIG’s 
responsibility  to  prepare  independent  estimates,  and  prompted  a  substantial  inerease  in  its 
staff.  22 

In  the  mid  1990s,  OSD  adopted  the  commereial  business  program  management 
device  of  integrated  product  teams  (IPTs)  for  overseeing  aequisition  programs. 23  This 


20  use,  Title  10,  section  2434. 

21  Donald  Smll,  ed..  The  Cost  Analysis  Improvement  Group,  A  History  (McLean,  Virginia:  Logistics 
Management  Institute,  1998),  pp.  32-33. 

22  From  about  10  people  to  about  30. 

23  According  to  the  Defense  Acquisition  University’s  Glossary:  Defense  Acquisition  Acronyms  and 
Terms  (12*  edition,  July  2005),  IPTs  are  “composed  of  representatives  from  appropriate  functional 
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involved,  inter  alia,  formation  of  cost  IPTs  to  promote  a  free  flow  of  information  among 
the  organizations  involved  in  program  costing  and  was  intended  to  foster  agreement  on 
methodologies  and  data  for  developing  cost  estimates  in  support  of  acquisition  milestone 

decisions  .24 

Data  on  cost  growth  in  the  pre-CAIG  era  are  scanty  and  not  of  good  quality,  but 
available  evidence  indicates  that  the  introduction  of  the  CAIG  and  independent 
parametric  cost  estimates  significantly  and  quickly  improved  the  quality  of  costing  and 
reduced  cost  growth.25  Data  from  more  recent  experience  have  not  yet  been  analyzed 
with  comparable  thoroughness,  but  there  are  indications  that  cost  growth  has  increased  in 
the  past  few  years. 

The  parametric  cost  estimating  process  critically  depends  on  the  quality  and 
breadth  of  the  data  about  actual  costs  from  which  it  can  draw.  For  this  reason,  DoD 
requires  submission  of  cost  data  in  a  form  suitable  for  analysis,  through  the  Cost  and 
Software  Data  Reporting  (CSDR)  system,  which  is  operated  by  the  CAIG. 26  The 
accuracy  of  a  parametric  estimate  of  a  system’s  cost  depends  on  clear  definition  of  the 
system’s  engineering  and  other  characteristics.  Estimates  of  development  costs  are 
inherently  less  accurate  than  estimates  of  production  costs. 

7.  Roles  in  Preparing  Cost  Estimates 

Program  cost  estimates  are  initially  prepared  by  PMOs,  assisted  by  Service  cost 
centers  and  inputs  from  contractors.  In  preparation  for  Milestone  B,  the  PM’s  cost 
estimate  is  reviewed  by  the  Service  cost  analysis  group,  and  after  being  modified  in  a 
give-and-take  process,  becomes  the  Service  Cost  Position  (SCP).  This  estimate  is  then 
reviewed  by  the  CAIG,  which  also  normally  develops  an  Independent  Cost  Estimate 


disciplines  working  together  to  build  successful  programs,  identify  and  resolve  issues,  and  make  sound 
and  timely  recommendations  to  facilitate  decision  making.  There  are  three  types  of  IPTs:  Overarching 
IPTs  (OIPTs)  that  focus  on  strategic  guidance,  program  assessment,  and  issue  resolution;  Working- 
level  IPTs  (WIPTs)  that  identify  and  resolve  program  issues,  determine  program  status,  and  seek 
opportunities  for  acquisition  reform;  and  Program-level  IPTs  (PIPTs)  that  focus  on  program  execution 
and  may  include  representatives  from  both  government  and  industry  after  contract  award.” 

24  Donald  Smll,  ed..  The  Cost  Analysis  Improvement  Group,  A  History,  pp.  41-43. 

26  Neil  M.  Singer,  “Cost  Growth  in  Weapon  Systems:  Recent  Experience  and  Possible  Remedies,” 
Congressional  Budget  Office,  October  12,  1982,  pp.  2-3;  Defense  Science  Board,  “Report  of  the 
Acquisition  Cycle  Task  Force,”  1977  Summer  Study,  15  March  1978,  pp.  68-9.  See  also  David  L. 
McNicol,  et  al.,  “The  Accuracy  of  Independent  Estimates  of  the  Procurement  Costs  of  Major 
Systems,”  Institute  for  Defense  Analyses,  IDA  Paper  P-3989,  August  2005. 

26  CAIG,  “Cost  and  Software  Data  Reporting  (CSDR)  Manual,”  DoD  5000.04-M- 1 ,  April  1 8,  2007. 
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(ICE)  .27  This  process  was  put  in  place  many  years  ago  beeause  it  was  clear  that 
reasonable  aecuracy  in  the  cost  estimate  was  essential  to  a  sound  deeision  on  the  cost- 
effeetiveness  of  the  proposed  program  and  whether  to  proeeed.  Independence  was 
considered  the  key  to  minimizing  the  Serviee’s  tendency  to  “buy  in”  by  underestimating 
the  likely  cost. 

Currently  it  is  DoD  policy  to  fund  programs  to  the  independent  CAIG  estimates 
(which  are  generally  higher  than  the  service  estimates)  unless  the  MDA  decides 
otherwise.  That  poliey  has  not  been  consistently  applied.  An  argument  frequently  made 
against  funding  to  a  higher  CAIG  estimate  is  that  the  higher  estimate  will  beeome  a  self- 
fulfilling  prophesy — that  it  is  better  to  “hold  the  program  manager’s  feet  to  the  fire”  to 
manage  his/her  program  within  the  SCP  estimate.  There  is,  however,  little  if  any 
empirieal  evidence  to  support  that  view,28  and  PMs  ean  be  incentivized  through  other 
means  to  pursue  the  lower  eost  goal.  Another  argument  that  is  made  from  time  to  time  is 
that  “this  program  is  different;”  e.g.,  “We  have  an  innovative  new  manufaeturing  proeess 
that  obviates  the  historical  data  on  which  parametric  production  cost  estimates  are  made.” 
Milestone  Decision  Authorities  have  often  been  persuaded  by  one  or  more  sueh 
arguments  and  have  approved  aequisition  baselines  less  costly  than  recommended  by  the 
CAIG. 

Studies  have  consistently  shown  that  produetion  costs  for  a  substantial  majority  of 
acquisition  programs  come  in  within  30%  of  the  estimates  made  at  Milestone  B  (or 
Milestone  II),  after  allowances  are  made  for  unforeseen  inflation,  quantity  ehanges,  and 
other  deliberate  program  ehanges.  In  this  group,  positive  and  negative  variances  from  the 
estimate  are  roughly  equally  distributed.  But  the  minority  of  programs  that  have 
produetion  cost  growth  of  more  than  30%  contribute  disproportionately  to  overall  cost 
growth. 29  Most  of  the  programs  studied  here  fall  or  are  projected  to  fall  into  the  latter 
category. 

On  the  whole,  growth  in  development  costs  is  proportionately  greater  than  in 
produetion  costs,  as  might  be  expected,  given  the  greater  uncertainties  in  the  “invention” 


27  Time  and  resources  permitting.  One  limitation  on  the  CAIG  review  process  is  late  submission  of  the 
SCP  and  the  accompanying  Cost  Analysis  Requirements  Description  (CARD),  which  is  supposed  to  be 
submitted  180  days  prior  to  the  milestone  decision  DAB. 

2^  McNicol,  et  al.,  “The  Accuracy  of  Independent  Estimates,”  IDA  Paper  P-3989,  pp.  11-13. 

29  For  a  particularly  comprehensive  analysis  see  David  L.  McNicol,  “Cost  Growth  in  Major  Weapon 
Procurement  Programs,”  Institute  for  Defense  Analyses,  IDA  Paper  P-3832,  October  2004,  pp.  18-19. 
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phase .30  However,  the  overall  impact  on  total  program  costs  is  usually  less  because 
RDT&E  constitutes  a  smaller  fraction  of  total  program  cost  than  procurement. 

8,  Acquisition  Program  Baselines 

Acquisition  Program  Baselines  (APBs)  are  a  legal  requirement  for  MDAPs  and 
must  be  approved  by  the  MDA.^i  The  PM  develops  a  draft  APB  that  sets  the  overall 
goals  of  the  program  including  specific  targets  for  performance,  cost,  and  schedule, 
generally  with  both  objective  values  and  thresholds  of  acceptability.  The  initial  APB 
must  be  prepared  for  approval  at  program  initiation  -  generally  the  normal  program 
initiation  milestone  shown  in  Figures  2  and  3,  although  some  programs  do  not  begin  or 
become  formally  recognized  until  a  later  stage  in  the  process.  New  APBs  are  required  at 
the  beginning  of  each  succeeding  phase.  The  PM  is  required  to  report  any  breach  of  an 
APB  threshold  to  the  MDA,  and  to  provide  plans  for  recovery  and  recommendations  for 
action.  Normally  this  process  results  in  a  revision  to  the  APB  to  reflect  the  new  plan. 
Problems  with  the  APB  process,  and  its  associated  Acquisition  Strategy  Reports,  are 
discussed  at  some  length  in  Chapter  II. 

9.  Affordability 

The  decision  to  start  an  acquisition  program  should  be  made  with  confidence  that 
the  resources  to  complete  the  acquisition  will  be  available.  Financial  resources  are 
allocated  through  the  DoD  Planning,  Programming,  Budgeting,  and  Execution  System 
(PPBES),  one  component  of  which  is  the  Department’s  Future  Years  Defense  Program, 
or  FYDP.  In  order  to  enter  EMD  at  MS  B,  DoDI  5000.02^^  requires  that  the  program  be 
fully  funded,  as  reflected  in  the  APB,  in  the  sponsoring  Component’s  Program  Objective 
Memorandum  33  (POM)  and  ultimately  in  the  FYDP.  (Problems  with  the  mapping  APBs 
to  the  FYDP  and  budget,  are  discussed  in  Chapter  II.) 

The  FYDP,  however,  only  contains  information  for  five  years  beyond  the  budget 
year.  Since  most  MDAPs  require  investment  funding  well  beyond  that  time  period,  an 


30  Ibid.,  p.  17. 

31  10  use  §2435. 

32  DoDI  5000.02,  Enclosure  2,  Engineering  and  Manufacturing  Development  (EMD)  Phase,  sec.  d  (4), 
(December  8,  2008),  p.23. 

33  Program  Objective  Memoranda  are  developed  biennially  by  each  DoD  Component  and  submitted  to 
the  Secretary  for  approval.  They  form  the  basis  for  DoD’s  next  budget  and  the  FYDP,  which  is  the 
mid-term  financial  plan. 
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important  issue  is  whether  sufficient  resources  to  complete  the  program  will  likely  be 
available  in  the  years  beyond  the  FYDP — that  is,  whether  it  will  be  affordable  in  the 
longer  term.  A  declaration  of  affordability  must  meet  three  conditions:  (1)  that  the  cost  of 
the  program  be  known  within  a  reasonable  level  of  certainty  (say,  within  15%);  (2)  that 
demands  for  funds  for  other  acquisition  programs  also  be  reasonably  well-defined;  and 
(3)  that  the  resulting  total  acquisition  costs  do  not  result  in  an  unbalanced  distribution  of 
funds  among  the  manpower,  operating,  and  acquisition  accounts. 

The  Defense  Program  Projection  (DPP)  is  a  process  that  periodically  examines 
such  longer  term  programmatic  issues.  It  is  based  on  approved  acquisition  and  major 
force  programs,  service  long-range  plans,  and  logical  extensions  of  acquisition  programs 
that  are  started  within  the  FYDP. 34  It  provides  the  primary  context  for  addressing  longer- 
term  acquisition  affordability  issues.35  It  is,  however,  seldom  used  in  the  acquisition 
process,  as  best  we  were  able  to  determine.  To  the  extent  that  affordability  assessments 
for  the  1 1  programs  were  formally  conducted,  they  considered  only  whether  the  required 
funds  would  be  available  in  the  FYDP. 

E.  DATA  SOURCES  AND  THEIR  QUALITY 

1.  Selected  Acquisition  Reports 

Selected  Acquisition  Reports  (SARs)  were  initiated  by  OSD  in  the  mid-1960s  to 
provide  basic  information  to  DoD  management  on  the  status  of  the  most  important 
acquisition  programs.  Since  1969,  Congress  has  required  that  the  SARs  for  MDAPs  be 
submitted  to  the  Congressional  defense  committees  within  60  days  of  submission  of  the 
President’s  Budget.  The  SARs  include  the  following  major  sections: 

•  Program  information: 

o  Responsible  Office:  Contact  information  for  the  program  manager  and 
Program  Executive  Officer  (PEO) 

o  Mission  and  Description:  An  overall  description  of  the  system  and  its 
mission 


34  The  usual  assumption  is  that  total  DoD  funding  will  be  constant  in  real  dollars  beyond  the  FYDP. 

33  The  DPP  analysis,  conducted  by  OSD(PA&E)  with  the  cooperation  of  the  Services,  other  DoD 
Components,  and  staff,  is  periodically  briefed  to  senior  DoD  leadership,  but  is  closely  held.  It  is  not  a 
formal  part  of  the  DoD  PPBES  process,  and  it  is  not  approved  by  the  Secretary.  When  the  DPP  was 
initiated  in  the  early  1990s,  the  AT&L  staff  participated  in  its  development;  however,  over  time, 
AT&L  active  participation  ceased. 
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o  Executive  Summary:  An  overview  of  current  status  by  the  program 
manager 

•  Threshold  Breaches:  Reports  any  breaches  to  schedule,  performance  or  cost 
thresholds  and  gives  reasons 

•  Schedule  and  Performance:  Objectives,  including  “objective”  and  threshold” 
values  for  schedule  and  performance  (each  a  separate  section) 

•  Track  to  budget:  Specifies  what  Program  Elements  in  the  DoD  budget  contain 
the  funding  for  the  program 

•  Cost  and  Funding: 

o  A  Summary  showing  total  acquisition  costs  (RDT&E,  Procurement,  and 
Military  Construction)  and  production  quantities,  and 
o  Annual  Eunding,  which  displays  RDT&E  and  procurement  costs  (by 
budget  appropriation)  and  procurement  quantities  by  year.  (Costs  are 
given  in  both  a  base-year  constant  dollars  and  inflated  (“then-year”) 
dollars.) 

•  Unit  Cost  Report:  Described  in  more  detail  below 

•  Cost  Variance:  Tracks  the  differences  in  costs,  with  a  table  comparing  the 
current  estimate  to  the  “baseline”  estimate  by  “reason”  for  the  cost  changes, 
followed  by  explanations  of  the  “reasons” 

•  Contracts:  Provides  names  and  addresses  of  contractors  and  dates  of 
contracts,  contract  costs,  (initial,  current,  and  estimate  at  completion),  and  cost 
and  schedule  variances 

•  Deliveries  and  Expenditures:  Gives  production  deliveries  and  funds  expended 

•  Operating  and  Support  Costs:  Provides  estimates  of  the  costs  to  operate  and 
support  the  system  after  fielding  and  compares  those  costs  to  a  comparable 
existing  system  (if  applicable). 

Most  of  these  sections  are  required  by  law  in  10  ElSC  Section  2432.  In  essence, 
the  SAR  is  a  periodic  report  on  the  progress  of  the  program,  with  the  greatest  emphasis 
on  unit  cost  measures.  The  comprehensive  history  of  SAR  cost  data  since  1970 
constitutes  the  primary  data  base  generally  used  for  studies  of  cost  growth.  Although  the 
SAR  data  base  suffers  from  significant  limitations  for  that  purpose,  there  are  no  good 
alternatives. 36  Because  the  SARs  are  so  important  to  Congress,  DoD  goes  to  greater 


36  Paul  G.  Hough,  “Pitfalls  in  Calculating  Cost  Growth  from  Selected  Acquisition  Reports,”  The  RAND 
Corporation,  N-3136-AF,  1992. 
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lengths  to  ensure  their  aeeuraey  than  it  does  in  maintaining  eomprehensive  internal 
reeords  of  MDAP  progress,  at  least  at  the  OSD  level.  This  study,  therefore,  largely  relied 
on  SAR  data  for  its  eost  growth  estimates. 

2,  Defense  Acquisition  Executive  Summaries 

The  office  of  the  USD(AT&L)  established  the  Defense  Acquisition  Executive 
Summary  (DAES)  system  to  meet  its  own  need  for  information.  Eor  the  most  part,  the 
DAES  reports  contain  the  same  information  as  SARs  but  they  are  updated  more 
frequently.  The  only  substantive  addition  in  the  DAES  is  an  “Assessments”  section 
provided  by  the  sponsoring  Component  (in  essence,  the  PM)  with  stoplight  charts  for 
“cost,  schedule,  performance,  funding  and  sustainment”  versus  the  APB  and  “contracts,” 
together  with  an  “Explanations”  section.  This  is  followed  by  a  section  entitled  “OSD 
Assessments”  for  the  same  topics.  In  our  review  of  the  data  for  the  1 1  programs  selected 
for  this  study,  only  occasionally  did  we  find  the  OSD  assessments  section  to  contain  any 
information. 

Originally,  DAES  reports  were  submitted  quarterly,  but  since  the  advent  of  a  web- 
based  acquisition  data  collection  system,  the  DAES  reports  are  supposed  to  be  updated  by 
the  Program  Management  Offices  (PMOs)  with  the  latest  available  program  information, 
once  it  becomes  “official.”  Our  study  did  not  include  a  formal  review  of  the  extent  to 
which  such  updates  are  being  provided. 

3,  Defense  Acquisition  Management  Information  Retrieval  System 

The  Defense  Acquisition  Management  Information  Retrieval  (DAMIR)  system  is 
an  internal  DoD  web-based  repository  of  SAR  and  DAES  reports,  with  historical 
documents  going  back  a  number  of  years.  The  following  description  is  from  the 

website:37 

DAMIR  is  a  DoD  initiative  that  provides  enterprise  visibility  to  Acquisition 
program  information.  The  primary  goal  of  DAMIR  is  to  streamline  acquisition 
management  and  oversight  by  leveraging  the  capabilities  of  a  net-centric 
environment.  DAMIR  identifies  various  data  sources  that  the  Acquisition 
community  uses  to  manage  Major  Defense  Acquisition  Programs  (MDAPs)  and 
Major  Automated  Information  Systems  (MAIS)  programs  and  provides  a  unified 
web-based  interface  through  which  to  present  that  information.  DAMIR  enables 
the  OSD,  Military  Services,  Congress  and  other  participating  communities  to 
access  information  relevant  to  their  missions  regardless  of  the  agency  or  where 


http://www.acq.osd.mil/damir/ 
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the  data  resides.  DAMIR  components  have  replaced  the  need  for  the  legacy 
Consolidated  Acquisition  Reporting  System  (CARS).^^ 

One  component  of  DAMIR,  Purview,  is  an  executive  information  system  that 
displays  program  information  such  as  mission  and  description,  cost,  funding  and 
schedule.  It  was  developed  under  the  DAMIR  initiative  to  provide  a 
comprehensive  view  of  the  current  state  of  all  MDAP  and  MAIS  programs.  It  is 
OSD's  solution  for  structured  acquisition  data  presentation  and  uses  web  services 
to  obtain  and  display  Defense  Acquisition  Executive  Summary  (DAES)  data 
directly  from  the  Service  acquisition  databases.  Within  Purview  users  can  also 
execute  Ad-hoc  reports.  In  addition,  select  DAMIR  users  can  create,  edit,  or 
review  the  following:  Selected  Acquisition  Reports  (SARs),  SAR  Baselines, 
Acquisition  Proposed  Baselines  (APBs),  and  Assessments. 

DAES  information  is  obtained  through  use  of  web  services  connected  to  Service 
acquisition  information  systems.  Some  insights  from  our  use  of  DAMIR  are  included  in 
the  next  chapter. 

4,  Earned  Value  Management  System 

The  Earned  Value  Management  System  (EVMS)  was  established  in  1967  as  the 
Cost/Schedule  Control  System  (C/SCS  or  (CS)^)  to  track  “earned  value”  (EV)  -  the  value 
of  the  work  done  on  an  acquisition  contract  relative  to  the  required  end  product  -  against 
actual  and  planned  expenditures.  If  expenditures  are  outrunning  EV,  it  is  clear  sign  of 
program  trouble,  and  the  degree  of  divergence  is  an  indicator  of  the  probable  extent  of 
ultimate  cost  growth.  Over  the  past  three  decades  the  responsibility  for  the  system  shifted 
from  the  DoD  Comptroller  to  the  USD(AT&E)  and  finally  to  the  Defense  Contracts 
Management  Agency  (DCMA).39  Accurate  EVMS  data,  properly  interpreted,  provides 
valuable  insights  into  program  status. 

5,  Nunn-McCurdy  Breaches  and  Unit  Cost  Reports 

The  law  requires  that  Congress  receive  “unit  cost  reports”  in  the  event  that  certain 
thresholds  of  cost  are  breached.'^o  These  are  generally  known  as  “Nunn-McCurdy 


DAMIR  has  a  multi-level  access  capability;  for  example,  DAES  reports  are  only  visible  by  DoD 
personnel,  and  some  portions  are  only  visible  by  users  explicitly  approved  by  OUSD(AT&L). 

^9  Government  Accountability  Office,  “Significant  Changes  Underway  in  DOD’s  Earned  Value 
Management  Process,”  GAO  /NSIAD-97-108,  May  1997. 

40  10  use  §2433. 
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breaches,”  after  the  sponsors  of  the  original  1982  law  (which  has  been  much  amended). 
The  cost-growth  thresholds  that  trigger  such  reports  are  shown  in  Table  2.^^ 


Table  2.  Nunn-McCurdy  PAUC  Reporting  Thresholds 


Significant  Breach 

Critical  Breach 

From  current  baseline 

+15% 

+25% 

From  original  baseline 

+30% 

+50% 

Each  Nunn-Mc Curdy  breach  requires  specific  DoD  management  and  reporting 
actions.  At  the  end  of  the  quarter  following  any  breach,  a  complete  SAR  must  be 
submitted,  with  an  explanation  of  the  reasons  for  the  cost  growth. 

In  addition,  when  the  breach  is  “critical,”  the  USD(AT&L)  must  certify  that: 

1 .  The  program  is  essential  to  the  national  security, 

2.  There  is  no  alternative  that  will  provide  equal  or  greater  capability  at  a 
lower  cost, 

3.  The  new  cost  estimates  are  reasonable,  and 

4.  The  management  structure  is  adequate  to  control  cost  growth. 

Under  the  law,  the  authority  to  obligate  funds  for  the  program  lapses  if  no 
certification  is  provided.  In  the  case  of  one  of  the  programs  we  studied,  the  Armed 
Reconnaissance  Helicopter,  the  USD(AT&L)  determined  that  he  could  not  make  such  a 
certification  and  terminated  the  program. 

When  a  program  is  certified  following  a  critical  Nunn-Mc  Curdy  breach,  a  new 
APB  is  required,  an  action  known  as  re-baselining.  Re -baselining  also  occurs  at  each 
subsequent  milestone  review.  Notwithstanding  the  establishment  of  a  new  “current 
baseline,”  the  values  set  out  in  the  Milestone  B  APB  must  continue  to  be  reported  as  the 
“initial  baseline.”  The  complete  baseline  history  is  contained  in  a  document  available  on 
DAMIR  for  MDAPs  past  Milestone  B/II. 

The  Nunn-McCurdy  breach  reports  have  proven  very  valuable  in  this  study  for 
understanding  the  causes  of  cost  growth. 


The  law  sets  identical  thresholds  for  growth  in  Average  Procurement  Unit  Cost  (APUC),  which  is 
obtained  by  dividing  total  procurement  costs  by  the  total  number  of  units  procured. 
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II.  FINDINGS  AND  CONCLUSIONS 


In  this  section  of  the  report,  we  present  our  principal  findings  and  conclusions 
regarding  the  causes  of  cost  growth  in  the  1 1  programs  examined.  The  chapter  begins  by 
dividing  the  major  causes  of  cost  growth  into  two  major  categories,  which  are  then  sub¬ 
divided  into  additional  categories.  Subsections  of  the  chapter  provide  explanation  and 
evidence  from  the  programs  examined  to  support  the  findings.  Finally,  we  identify  three 
lesser  causes,  which  are  presented  and  explained  in  subsequent  subsections.  Appendices 
for  each  of  the  1 1  programs  present  its  history  and  explore  in  greater  depth  the  reasons 
for  its  cost  growth.  While  some  of  that  material  has  been  brought  into  this  chapter  to 
support  our  findings,  brevity  dictated  that  much  was  not  used,  so  the  reader,  if  privy  to 
FOUO  material,  is  encouraged  to  explore  the  appendices  in  Volume  3  for  additional 
important  insights. 

A  caveat:  Our  findings  are  based  on  the  examination  of  the  1 1  programs  which,  as 
was  stressed  Chapter  I,  were  selected  because  of  their  significant  cost  growth.  We  did  not 
examine  the  many  successful  DoD  acquisition  programs  that  did  not  have  large  cost 
growth.  But  even  if  these  findings  and  conclusions  were  found  to  apply  exclusively  to  the 
1 1  programs  (an  extremely  unlikely  premise),  they  deserve  serious  consideration  because 
of  the  importance  of  those  particular  programs. 

Avoiding  cost  growth  is  not  just  a  narrowly  technical  matter  of  having  the  right 
organizational  structure  staffed  by  capable  professionals  using  sound  policies  and 
procedures.  Indeed,  generally  sound  policies  and  procedures  have  long  been  in  place. 
Decisions  on  starting  a  new  major  defense  acquisition  program  involve  making  choices 
on  critically  important  matters  of  national  security  and  huge  expenditures  of  resources. 
Accordingly,  the  key  decisions  on  MDAPs  are  made  by  politically  accountable  officials. 
Good  outcomes  require  that  those  officials  be  given  an  accurate  assessment  of  the 
decisions  to  be  made  and  sound  advice  on  the  consequences  of  various  alternatives.  There 
is,  however,  also  a  legitimate  policy  dimension  to  their  decisions  in  that  the  top  officials 
of  DoD  decide,  explicitly  or  implicitly,  how  much  risk  should  be  borne  in  acquisition 
programs,  and  those  choices  in  turn  have  an  important  influence  on  subsequent  cost 
growth. 
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A.  OVERVIEW  OF  COST  GROWTH  MAJOR  CAUSES 


Figure  4  summarizes  the  major  eauses  we  identified  through  our  analysis  of 
growth  in  our  sample  of  high-growth  MDAPs. 

1.  Weaknesses  in  management  visibility,  direction,  and 
oversight 

a.  Lax  or  inappropriate  implementation  of  policies 

b.  Excessive  reliance  on  unproven  management  theories  and 
acquisition  strategies 

c.  Poor  contractor  selection,  oversight,  and  incentivization 

2.  Weaknesses  in  initial  program  definition  and  costing 

a.  Defective  and  unstable  requirements  processes 

b.  Entry  into  development  with  immature  technologies 

c.  Deficient  front-end  analysis  of  system-level  design  issues  and 
technical  risks 

d.  Excessive  schedule  compression  and  concurrency 


Figure  4.  Major  Causes  of  Acquisition  Cost  Growth 

As  will  be  seen,  no  single  major  eause  explains  eost  growth  for  the  majority  of  the 
programs.  The  eauses  listed  in  Figure  4  are  not  mutually  exelusive.  Many  are  elosely 
interrelated,  and  there  are  eompounding  effeets  in  whieh  the  various  eauses  of  eost 
growth  interaet  and  reinforee  eaeh  other.  For  example,  when  a  program  reeeives  approval 
to  enter  development,  despite  being  so  ill-defined  and  teehnieally  immature  that  its  eosts 
eannot  be  estimated  with  even  rough  eonfidenee,  it  presents  prima  facie  evidenee  not 
only  of  weaknesses  in  management  oversight,  ineluding  lax  implementation  of  polieies, 
but  also  in  program  definition  and  eosting.  For  this  reason,  we  eannot  assign  speeifie 
pereentages  of  eost  growth  to  individual  eauses.  Nevertheless,  our  analysis  indieates  that, 
while  the  eauses  manifest  themselves  in  different  ways  from  program  to  program,  there 
are  important  eommonalities.  And  when  taken  together,  these  faetors  tend  to  dominate, 
forming  the  basis  for  the  reeommendations  presented  in  the  next  ehapter. 

Table  3  doeuments  the  eost  growth  in  the  11  programs  from  the  initial  APB 
through  eaeh  re -baselining  to  the  eurrent  estimate  as  refieeted  in  the  latest  DAES  report. 
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Table  3.  Cost  History  of  Programs  Examined 


APB  estimate  at  MS-B 

First  re-basetining 

Second  re-baselining 

Third  re-baselining 

Current  Estimate 

Date 

Date 

Date 

ARH 

RDT&E($B) 

0.94 

Note  1 

Procurement($B) 

4.60 

6.3 

PAUC  ($M) 

10.3 

1 

14.15 

APUC($M) 

9.0 

12.31 

Qty 

512 

512 

FCS 

RDT&E($B) 

18.0 

26.44 

Nov-05 

27.0 

Procurennent{$B) 
PAUC  ($M) 
APUC($M) 

59.1 

92.79 

98.4 

WIN-T 

RDT&E($B) 

0.71 

1.40 

Jun-07 

Procurement{$B) 

9.15 

11.72 

PAUC  ($M) 

9.9 

13.1 

17.2 

Note  2 

APUC($M) 

9.2 

11.7 

JTRS-GMR 

RDT&E($B) 

0.85 

1.21 

Jan-08 

1.43 

Procurement{$B) 

13.59 

13.06 

12.27 

PAUC  ($M) 

0.133 

0.165 

0.158 

APUC($M) 

0.126 

0.151 

0.142 

Qty 

108086 

86652 

86652 

Global  Hawk 

RDT&E($B) 

0.84 

■dtiM 

Mar-02 

Dec-02 

3.08 

Sep-07 

3.00 

Procurement{$B) 

3.48 

3.76 

2.91 

4.91 

5.09 

PAUC  ($M) 

69.0 

117 

102 

150 

151.6 

APUC($M) 

55.0 

74 

57 

91 

94.3 

Qty 

63 

51 

51 

54 

77 

JSF 

RDT&E($B) 

32.3 

42.1 

Mar-04 

42.1 

Mar-07 

40.2 

Procurement{$B) 

143.3 

149.5 

169.0 

169.4 

PAUC  ($M) 

61.8 

78.6 

86.5 

85.5 

APUC($M) 

50.2 

61.2 

69.2 

69.3 

Qty 

2842 

2443 

2443 

2443 

JASSM 

RDT&E($B) 

0.77 

0.89 

Dec-01 

1 

1.07 

Dec-02 

1.10 

Mar-04 

1.15 

Procurement{$B) 

0.96 

1.62 

1.87 

2.90 

4.06 

PAUC  ($M) 

0.709 

0.669 

1 

0.779 

0.737 

1.16 

APUC($M) 

0.400 

0.439 

0.509 

0.542 

0.686 

Qty 

2400 

3700 

3700 

5353 

5353 

LPD-17 

RDT&E($B) 

0.08 

0.09 

May-97 

0.11 

Dct-05 

0.11 

May-07 

0.12 

Procurement{$B) 

8.94 

8.93 

12.84 

10.99 

14.37 

PAUC  ($M) 

751.6 

751.6 

1079.6 

1079.6 

1316.5 

APUC($M) 

743.8 

743.8 

1070.2 

1070.2 

1305.9 

Qty 

12 

12 

12 

9 

10 

LCS 

RDT&E($B) 

1.19 

2.23 

2.95 

Note  3 

Procurement{$B) 
PAUC  ($M) 
APUC($M) 

EFV 

RDT&E($B) 

1.53 

1.78 

Nov-02 

Mar-05 

3.48 

Aug-07 

3.83 

Procurement{$B) 

6.82 

7.35 

8.92 

9.62 

10.58 

PAUC  ($M) 

8.2 

8.9 

11.1 

22.2 

17.5 

APUC($M) 

6.7 

7.3 

8.8 

16.8 

17.3 

Qty 

1025 

1025 

1025 

593 

574 

SBIRS 

RDT&E($B) 

3.33 

3.02 

Mar-98 

5.43 

Sep-02 

7.02 

Mar-06 

7.43 

Procurement{$B) 

0.50 

1.26 

1.34 

2.20 

PAUC  ($M) 

694 

736 

1468 

2856 

2455 

APUC($M) 

248 

421 

1343 

1100 

Qty 

5 

5 

5 

5 

4 

Note  1:  Not  rebaselined.  First  “rebaselining  ”  entry  reflects  Sep-07  SAR.  “Current  estimate  ”  values  are  those 
estimated  by  CAIG  at  the  Nunn-McCurdy  Breach 
Note  2:  Estimate  at  the  Nunn-McCurdy  Breach 

Note  3:  LCS  costs  are  based  on  President’s  Budget  (PBP  data,  not  SARs,  since  the  SARs  costs  do  not  include  all 
subsystems.)  The  three  costs  are  from  the  FY  2004,  FY  2008,  and  FY  2009  PBs  respectively. 
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1,  Weaknesses  in  Management  Visibility,  Direction,  and  Oversight 

This  broad  category  focuses  on  a  general  lack  of  discipline  and  attention  to  detail 
among  the  offices  that  have  been  responsible  for  overseeing  MDAPs  in  recent  years. 
Evidence  of  this  general  lack  of  attention  includes  the  decline  in  the  documentation  of 
independent  program  assessments  in  the  DAES.  The  DAES  process  was  originally 
intended  to  provide  the  DAE  with  timely  warning  of  problems,  based,  in  part,  on 
systematic  evaluations  of  earned  value.  Other  evidence  of  weakness  includes  the  uneven 
quality  of  the  directions  provided  in  Acquisition  Decision  Memoranda  and  their 
associated  Acquisition  Strategy  Reports,  as  well  as  the  lack  of  a  system  for  tracking 
compliance  with  the  DAE’s  decisions,  including  the  tracking  of  funding  decisions  into 
subsequent  DoD  budget  requests.  One  result  of  this  general  lack  of  discipline  was  that 
problems  in  some  programs  festered  for  years  with  no  formal  DAE  review  until  legal 
thresholds  were  breached. 

On  the  other  hand,  several  of  the  programs  received  quite  a  lot  of  management 
attention  (e.g.,  ECS,  Global  Hawk,  JSE,  JTRS-Ground  Mobile  Radio  (GMR),  and 
SBIRS),  but  top-level  guidance  did  not  prove  effective  in  getting  troubled  programs  on 
track.  ECS  and  Global  Hawk,  despite  annual  DAB  reviews,  continued  to  have  major 
problems.  The  reasons  for  these  shortfalls  are  complex,  and  much  of  the  fault  probably 
lies  within  the  DoD  culture  of  optimism.  When  a  large-scale  program  gets  off  to  a  bad 
start  or  heads  off  in  the  wrong  direction,  it  is  very  difficult,  for  cultural  reasons,  for  a 
DAE  to  take  the  kind  of  strong  action,  such  as  major  program  restructuring  or 
cancellation  that  may  be  required.^2 

Even  when  major  restructuring  is  undertaken,  it  will  not  be  effective  if  the 
fundamental  problem  is  not  addressed.  A  case  in  point  is  JTRS-GMR.  In  2006,  the  DAE 
directed  a  major  restructuring  of  the  program,  but  failed  to  address  the  underlying  issue 
that  the  technology  was  simply  not  mature  enough  to  accomplish  the  program’s 
objectives. 

One  cause  of  poor  decision-making  is  the  lack  of  good  information.  The  next  two 
sub-sections  address  our  finding  that  in  the  last  few  years  there  has  been  a  deterioration  in 
the  systems  that  provide  information  on  programs  to  top  acquisition  leadership.  Except 
for  certain  high  visibility  programs,  the  DAE  does  not  routinely  review  programs  that  do 


These  complex  reasons  are  beyond  the  scope  of  this  study. 
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not  appear  to  be  in  trouble.  But  as  noted,  the  systems  for  revealing  timely  evidence  that 
programs  are  in  trouble  have  been  working  badly.  When  we  asked  people  who  had  served 
as  DAEs  during  periods  when  signs  of  trouble  were  emerging  in  specific  MDAPs 
whether  they  could  recall  having  been  alerted  to  them  their  answers  were  largely 
negative. 

The  Defense  Acquisition  Executive  Summary 

As  discussed  in  the  introduction,  the  DAES  system  was  put  in  place  two  decades 
ago  to  provide  periodic  visibility  into  a  program’s  progress  and  a  timely  warning  of 
developing  problems.  In  theory,  every  active  MDAP  is  considered  for  review  once  a 
quarter.  Among  the  score  or  more  of  programs  up  for  review  each  month,'^3  ^  few  are 
selected  by  the  AT&E  OSD  staff — sometimes  over  Service  objections  -  for  discussion  at 
a  DAES  meeting — at  least  nominally  with  the  DAE.  Elnfortunately,  this  system  has 
become  ineffective  in  recent  years.  In  the  early  2000s,  responsibility  for  the  DAES  was 
moved  from  a  directorate  that  has  historically  taken  a  relatively  objective  view  of 
programs  to  a  directorate  that  is  more  closely  associated  with  program  proponency. 
Moreover,  some  recent  senior  executives  have  not  seen  the  value  in  the  process,  have  not 
attended  the  monthly  meetings,  and  have  not  encouraged  the  AT&E  staff  (aided  by  the 
rest  of  the  OSD  staff  and  Joint  Staff)44  to  bring  forth  potential  problems  for  review. 

As  previously  mentioned,  the  DAMIR  system,  the  repository  for  both  SAR  and 
OSD  DAES  reports,  is  linked  to  Component  acquisition  information  systems  in  order  to 
provide  timely  updates  of  program  information.  But  like  any  system,  DAMIR  is  only 
useful  if  the  data  are  current  and  used  for  analysis.  Unfortunately,  component  program 
managers  have  little  incentive  to  reveal  problems  when  they  first  threaten  to  arise  since  it 
is  human  nature  is  to  wait  and  see  if  the  problem  can  be  solved  without  the  attention  of 
higher  echelons  (a  “strategy  of  hope”).  Our  review  indicated  that  periodic  OSD 
assessments  have  only  infrequently  been  conducted  and  included  in  the  DAMIR  in  recent 
years.  Thus,  emerging  problems  appear  to  have  rarely  been  identified  to  the  DAE  via  the 
DAMIR  system. 


43  There  are  approximately  100  MDAP  and  pre-MDAP  programs  that  are  subjeet  to  review.  (Pre-MDAPs 
are  programs  that  will  likely  qualify  as  MDAPs  onee  they  pass  MS  B.) 

44  OSD  staff  elements  sueh  as  CAIG,  Director,  Operational  Test  &  Evaluation  (OT&E),  and  AT&E 
Systems  and  Software  Engineering  frequently  learn  of  difficulties  in  programs  through  their  own 
channels  of  information.  If  incorporated  into  DAES,  these  inputs  could  provide  valuable  and  timely 
information  to  the  DAE  on  emerging  problems. 
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Recently  Congress  has  recognized  the  need  for  improved  independent 
assessments  of  the  status  of  major  acquisition  programs  and  has  included  a  requirement 
for  a  new  Performance  Assessment  and  Root  Cause  Analysis  (PARCA)  process  in  the 
WSARA.45  It  is  expected  that  the  office  assigned  responsibility  for  PARCA  will  establish 
a  review  process  that  either  revitalizes  or  replaces  the  somewhat  dysfunctional  DAES. 

There  are  several  elements  of  a  “proper”  periodic  review  of  program  status.  These 
include  the  achievement  of  specific  contract  and  APB/ASR/ADM  milestones,  such  as 
first  test  article  delivery  and  a  host  of  more  detailed  milestones  set  forth  in  approved 
program  plans.  Such  reviews  would  require  the  OSD  PARCA  staff  to  systematically 
track  actuals  against  such  plans  including  promises  made  at  the  time  of  program  initiation 
or  at  subsequent  reviews.  The  lack  of  an  OSD  system  for  independently  tracking  such 
plans  and  promises  is  addressed  in  our  recommendations  chapter. 

EVMS  and  CSDR 

In  addition  to  tracking  programs  against  discrete  milestones,  the  oversight 
function  is  intended  to  benefit  from  the  EVMS.  The  primary  purpose  of  the  EVMS  is  to 
help  program  managers  track  their  own  progress  against  their  own  plans  as  reflected  in 
the  work  breakdown  structures  in  their  cost-type  contracts.  EVMS,  together  with  the 
CSDR  system,  were  also  intended  to  provide  valuable  information  to  senior  management. 
In  practice,  those  systems  frequently  lack  rigor  and  fail  to  provide  the  insight  into  actual 
program  progress  that  they  could. 

In  the  case  of  ARH,  clear  warning  signs  in  the  EVMS  that  a  massive  contract  cost 
overrun  was  in  train  were  ignored — reportedly  because  “everyone  knew  Bell  Helicopter’s 
EVMS  was  defective.”  (It  had  in  fact  been  de-certified.)  Nonetheless,  in  this  case  the 
EVMS’s  indications  of  a  serious  cost  overrun  were  accurate.  As  part  of  streamlining 
efforts  under  acquisition  reform,  contractors  were  sometimes  allowed  to  report  cost  and 
schedule  performance  in  their  own  formats.  Such  was  the  case  with  JASSM  and  SBIRS. 
The  result  for  JASSM  rendered  EVMS  “not  reliable  for  management  use,”  according  to 
the  Nunn-McCurdy  certification  assessment  by  DCMA,  while  a  “leaned-out”  program 
office  failed  to  take  notice.  Those  practices  have  often  impeded  visibility  into  developing 
problems,  as  they  did  in  SBIRS  and  JASSM. 


Weapons  Systems  Aequisition  Reform  Aet  of  2009,  Seetion  103. 
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ADMs  and  other  DAE  memoranda  frequently  contain  specific  direction  for 
program  content,  often  intended  to  help  contain  costs.  Since  there  is  no  system  to  track 
whether  or  how  well  such  direction  is  carried  out,  we  were  unable  to  determine  whether 
failure  to  execute  DAE  direction  contributed  to  cost  growth. 

Other  Evidence  of  Top-Level  Oversight  Weaknesses 

As  we  have  already  noted,  the  early  signs  of  trouble  in  the  ARH  revealed  by  the 
EVMS  were  not  heeded.  And  four  and  a  half  years  elapsed  between  ADMs  for  the  ECS, 
during  which  both  schedule  and  cost  grew  by  more  than  100%.  JASSM  was  re¬ 
designated  as  AC  AT  1C  for  Air  Eorce  oversight  after  completion  of  development,  in  the 
name  of  acquisition  reform  and  streamlining.  Over  the  next  five  years  serious  reliability 
problems  emerged.  Average  Procurement  Unit  Cost  (APUC)  grew  by  more  than  45%,  the 
program  was  expanded  to  include  a  new,  more  expensive  extended  range  variant,  and 
planned  procurement  quantity  doubled  without  JROC  approval.  We  did  not  find  any 
evidence  that  these  issues  received  any  serious  attention  from  the  OUSD(AT&E)  until 
recurring  missile  reliability  problems  resulted  in  cost  growth,  and  a  critical  Nunn- 
McCurdy  breach. 

a.  Lax  or  Inappropriate  Implementation  of  Policies 

Over  the  years  the  Department  has  accumulated  a  great  deal  of  experience  on  how 
to  establish  and  manage  sound  major  acquisition  programs.  These  “lessons  learned”  track 
back  to  and  before  the  1986  Packard  Commission  reporU^  and  are  well  codified  in 
standing  DoD  policies  and  procedures.  They  can  be  summarized  as  “Do  it  right  the  first 
time.”  These  “standing  orders”  intentionally  provide  enough  flexibility  to  allow  each 
unique  program  to  be  sensibly  structured  and  managed  without  imposing  an  inefficient 
straightjacket.  One  way  to  interpret  the  findings  in  this  report  is  to  observe  that  many  of 
the  problems  it  identifies  can  be  attributed  to  excessive  use  of  that  flexibility.  Indeed,  it  is 
common  for  outside  critics  of  defense  acquisition  practices  to  say  in  effect  “These 
problems  wouldn’t  have  happened,  or  wouldn’t  have  been  so  serious,  if  the  Department 
had  only  followed  its  own  policies.” 

More  specifically,  we  identified  several  standing  policies  whose  neglect 
contributed  seriously  and  directly  to  the  problems  encountered  by  several  of  these 


A  Quest  for  Excellence:  Final  Report  to  the  President  by  the  President’s  Blue  Ribbon  Commission  on 
Defense  Management,  June  1986. 
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MDAPs.  Two  of  those  policies  were  of  key  importance:  (1)  ensuring  that  the 
requirements  for  a  new  acquisition  program  are  well-understood  and  firm;  and  (2) 
ensuring  that  technologies  critical  to  successful  full-scale  development  and  production  of 
the  system  are  sufficiently  mature.  Both  those  issues  plagued  many  of  the  programs  we 
examined,  and  will  be  discussed  in  more  detail  in  subsequent  sections  of  this  chapter. 

Most  existing  policies  governing  MDAPs  were  adopted  in  response  to  lessons 
learned  from  program  failures,  with  the  implication  that  adherence  to  them  is  necessary  to 
avoid  repetition  of  such  failures.  Nevertheless,  DoD  has  repeatedly  waived  or  ignored  the 
application  of  policies,  or  relaxed  policies  without  analysis  of  the  implications. 

New  Program  Starts 

The  Introduction  reviewed  the  history  of  DoD’ s  search  for  the  right  way  to  start 
acquisition  programs.  Responding  to  the  reality  that,  at  least  in  some  circumstances,  it  is 
not  necessary  to  “start  from  scratch,”  DoD  implemented  more  flexible  options  for  starting 
programs  in  the  2003  version  of  DoDI  5000.2.47  Programs  were  allowed  to  come  to  the 
DAB  for  approval  to  enter  formal  MDAP  status  directly  at  MS  B  or  even  Milestone  C, 
thus  bypassing  the  earlier  decision  points.  Out  of  the  eleven  MDAPs  in  our  sample,  five 
did  not  have  MS  A  reviews,  but  the  justifications  for  failing  to  do  so  now  appear  dubious. 
In  the  case  of  Global  Hawk,  one  might  argue  that  bypassing  MS  A  was  justified  because 
the  program  was  a  transition  from  a  DARPA^s  Advanced  Concept  Technology 
Demonstration  (ACTD)  program,  which  includes  some  of  the  features  of  a  Technology 
Development  phase.  However,  we  were  informed  by  knowledgeable  people  that  in  fact 
the  ACTD  aircraft  was  not  ready  for  full-scale  development  and  significantly  deficient  in 
systems  design  and  engineering.  Problems  traceable  to  that  poor  start  have  plagued  the 
program  ever  since.  Almost  all  of  the  1 1  programs  suffered  serious  problems  that  relate 
to  the  lack  of  front-end  review  and  adequate  technology  development. 

The  PCS  program  is  another  salient  example.  The  program  was  initiated  by  fiat 
from  the  Chief  of  Staff  of  the  Army,  who  was  reacting,  in  part,  to  the  difficulties  the 


47  This  change  in  policy  entailed  risks:  acquisition  programs  are  supposed  to  be  initiated  in  response  to 
needs  to  provide  military  capabilities  to  enable  accomplishment  of  national  security  goals  and 
objectives,  as  codified  in  the  President’s  National  Security  Strategy  and  the  Secretary  of  Defense’s 
National  Defense  Strategy.  The  “front  end”  of  the  acquisition  process  is  supposed  to  ensure  this  aim. 
Thus,  to  short-change  the  front  end  is  to  risk  that  a  program  being  initiated  is  in  fact  not  consistent  with 
priorities  expressed  or  implied  by  those  basic  documents. 

48  Defense  Advanced  Research  Projects  Agency. 
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Army  experienced  in  deploying  a  heavy  combat  brigade  from  Germany  to  Kosovo  in 
1999.  Without  evidence  of  in-depth  supporting  analysis,  the  Chief  concluded  that  the 
Army  needed  to  build  a  much  lighter  force  that  would  still  be  capable  of  defeating 
traditional  heavy  forces  through  the  use  of  superior  battlefield  knowledge.  He  requested 
that  DARPA  undertake  development  of  several  of  the  key  technologies  needed  to 
implement  the  concept.  In  2002,  shortly  before  his  retirement,  the  Chief  of  Staff  directed 
the  Army  to  take  the  program  to  the  DAB  for  a  MS  B  decision,  and  that  review  was  held 
in  May  2003.  There  was  no  MS  A  or  earlier  USD  (AT&L)  approval  process  for  this 
massive  new  program 

The  omission  of  MS  A  was  particularly  damaging  because  of  the  world  situation 
at  the  time.  The  terrorist  attacks  of  September  11,  2001  and  ensuing  events  had 
fundamentally  changed  the  national  security  environment  and  thus  national  defense 
priorities.  Was  a  multi-billion  dollar  program  to  achieve  a  quantum  improvement  in  the 
capabilities  of  Army  heavy  forces  for  conventional  warfare  consistent  with  the 
capabilities  the  future  Army  would  need?  We  found  no  evidence  that  this  key  question 
was  asked.49  And  given  the  high  level  of  technical  risk  and  uncertainty  with  the  attendant 
cost  risk,  would  the  Army  be  able  to  afford  the  program  in  the  long  run  without 
detracting  from  capabilities  the  Army  would  need  to  fight  the  global  “war  on  terrorism?” 
Again,  apparently  these  questions  were  not  asked,  or  if  asked,  they  were  not  given 
sufficiently  serious  consideration. 

Even  if  a  program  initially  passes  the  test  of  consistency  with  national  defense 
priorities,  the  question  must  be  re-examined  throughout  the  life  of  the  program  because 
major  acquisition  programs  tend  to  stretch  over  many  years,  during  which  national 
security  objectives  can  change.  That  is  precisely  what  happened  for  several  of  the 
programs  that  we  examined,  including  the  expeditionary  fighting  vehicle  (EFV),  LPD-17 
and,  less  clearly,  the  short  takeoff/vertical  landing  (STOVE)  capability  for  JSE.  The 
continuation  of  these  programs  can  be  questioned  on  the  basis  that  opposed  amphibious 
assaults  have  considerably  less  utility  in  the  post-9/ 11  national  security  environment. 
Inclusion  of  the  STOVE  requirement  for  JSE  for  the  United  Stated  Marine  Corps 


We  realize,  of  eourse,  that  it  is  possible  that  the  question  eould  have  been  raised  without  leaving 
diseoverable  evidenee. 
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(USMC),  which  had  (and  is  having)  a  major  impact  on  the  cost  growth  in  the  program, 
was  deeided  without  apparent  serious  eonsideration  of  this  issue. 

SBIRS  also  avoided  MS  A  examination  (or  Milestone  I,  as  it  then  was).  There  had 
been  prior  attempts  to  initiate  programs  to  fill  the  same  global  infrared  surveillance  need 
which  had  foundered  on  the  same  requirements  and  cost  issues  that  have  so  troubled 
SBIRS,  and  it  is  clear  that  evading  consideration  of  these  issues  at  a  MS  A  review  was 
not  sound. 

Neither  WIN-T  nor  JTRS-GMR  had  a  MS  A  review.  We  have  not  uncovered  the 
reasons  why,  other  than  a  general  belief  that  eommercial  communications  technology  was 
well  advanced  and  applicable  to  these  needs,  but  we  found  no  evidenee  that  the  MS  A 
question  was  ever  explicitly  considered.  The  state  of  teehnology  readiness  elearly  did  not 
support  entry  into  System  Development  and  Demonstration  (SDD)  (see  next  section).  An 
MS  A  review  is  exactly  what  should  have  transpired,  in  lieu  of  an  MS  B.  It  would  have 
been  a  forum  for  closer  examination  of  what  proved  to  be  a  deeply  flawed  and 
fundamental  assumption. 

b.  Excessive  Reliance  on  Unproven  Management  Theories  and  Acquisition 

Strategies 

Most  of  these  troubled  programs  had  been  subjeeted  in  one  way  or  another  to 
various  “aequisition  reform”  initiatives  that  have  a  long  and  somewhat  eheckered  history 
since  their  high  point  as  key  elements  of  the  Clinton  Administration’s  “Reinventing 
Government”  initiative.  In  the  early  2000s,  “aequisition  transformation”  was  part  of  the 
overall  business  transformation  initiative. 

Many  of  those  initiatives  grew  out  of  pereeptions  in  industry  and  some  business 
sehools  that  the  Department  was  needlessly  slowing  aequisition  programs  and  driving  up 
costs  by  imposing  excessive  speeifications,  compliance  rules,  and  testing  requirements.  If 
only  DoD  would  aequire  new  weapons  systems  more  like  the  private  sector,  the  theory 
went,  then  new  equipment  would  get  into  operational  inventories  faster  and  at  lower  cost 
to  the  taxpayer.  Usually  overlooked  in  sueh  exhortations  was  the  fact  that  the  private 
sector  has  very  little  experience  in  developing  systems  as  large,  eomplex,  or 
teehnologioally  ehallenging  as  the  typical  MDAP.  When  technically  complex  and 


Another  consideration  with  regard  to  STOVL  was  the  United  Kingdom  requirement  for  a  Harrier 
replacement.  According  to  one  senior  official  that  consideration  was  actually  the  driver. 


32 


challenging  commercial  systems  are  developed,  it  is  almost  always  “in  house,”  with  the 
details  concealed  from  public  view.  Such  work  generally  constitutes  the  core  competence 
of  big  industrial  companies  and  is  not  “outsourced”  under  contracts  to  outside  companies 
as  are  DoD  MDAPs.  The  Boeing  787  “Dreamliner”  is  unusual  in  that  its  internal  troubles 
have  come  at  least  partly  into  public  view,  but  in  fact  such  cost  growth  and  delays  are 
common  in  highly  complex  private  sector  programs,  just  as  they  are  in  MDAPs. 

Most  of  the  programs  we  looked  at  began  during  those  two  decades,  and  the 
reform  efforts  generally  failed  to  deliver  the  promised  results,  in  fact  more  often 
damaging  the  programs  to  which  they  were  applied.  Of  particular  note,  highly  optimistic 
assumptions  about  the  cost-saving  benefits  of  acquisition  reform  initiatives  that  were 
reflected  in  early  cost  estimates  have  proven  to  result  in  high  cost  growth  when  the 
reforms  failed  to  deliver.  When  such  an  outcome  is  a  clear  risk  in  the  assumptions  used  in 
the  service  cost  estimate,  the  CAIG  should  reasonably  be  expected  to  raise  a  large 
warning  flag  (as  in  fact  it  clearly  did  for  PCS). 

In  some  cases  the  novel  policies  appear  to  have  been  adopted  in  hopes  of 
accommodating  external  constraints  on  schedules  or  budgets,  again  without  clear 
evidence  that  they  were  likely  to  be  effective.  In  order  to  speed  up  the  acquisition 
process,  prudent  early  stages  of  program  development  and  review  were  bypassed,  and 
full-scale  development  or  even  Low-Rate  Initial  Production  (LRIP)  were  initiated  with 
inadequate  levels  of  technology  readiness. 

Two  reform  initiatives  that  have  played  particularly  strong  roles  in  the  programs 
we  examined  were  “Cost  as  an  Independent  Variable”  (CAIV)  and  Total  System 
Performance  Responsibility  (TSPR).  Under  CAIV,  once  a  well-defined  and  understood 
cost  estimate  for  a  program  is  decided  on,  the  program  manager  should  be  tasked  to 
manage  to  achieve  that  cost — i.e.,  cost  is  “independent” — not  dependent  on  other 
program  parameters.  While  seemingly  a  logical  way  to  control  costs,  the  approach  is 
fraught  with  difficulties,  not  the  least  of  which  is  that  of  determining  a  sound  cost  figure 
at  the  beginning  of  full-scale  development.  Another  difficulty  of  long  standing  is  the 
general  reluctance  to  give  the  program  manager  sufficient  authority  to  make  the  cost- 
performance  trade-offs  that  are  essential  to  a  successful  CAIV  approach  -  even  within  the 
ranges  established  for  the  Key  Performance  parameters  (KPPj.^i  TSPR  was  an  Air  Force 


This  is  usually  not  a  simple  matter.  Requirements  tradeoffs  should  be  made  within  a  proeess  of  eareful 
eoordination  between  the  developer  and  the  user. 
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initiative  to  provide  greater  overall  responsibility  to  the  contraetor  to  execute  the 
program,  within  wide  latitudes,  with  a  marked  reduction  in  government  oversight 
compared  to  previous  programs.  Several  programs  among  our  group  suffered  from  the 
application  of  these  two  initiatives,  including  JASSM,  Global  Hawk,  SBIRS,  and  LCS. 

A  common  theme  expounded  by  many  proponents  of  acquisition  reform  was  to 
reduce  the  government’s  perceived  micromanagement  of  the  defense  industry,  which  also 
provided  a  rationale  for  reducing  the  DoD’s  supervisory  acquisition  workforce. ^2  Some 
program  management  offices  become  little  more  than  skeletons.  Contractors  were 
granted  expanded  powers  and  many  reporting  requirements  were  reduced  or  eliminated 
with  no  effort  to  fund  expanded  management  and  systems  engineering  by  the  contractors 
to  compensate  for  the  cuts  in  government  input.  For  the  programs  in  our  sample,  these 
initiatives  made  significant  contributions  to  cost  growth  and  schedule  delays,  in  large 
part,  because  they  prevented  early  detection  and  correction  of  problems. 

The  JASSM  missile  program  was  a  virtual  poster  child  of  acquisition  reforms 
popular  during  the  mid-  to  late  1990s.  As  described  in  considerable  detail  in  our  JASSM 
appendix,  a  veritable  cocktail  of  reform  initiatives  were  applied  to  the  program  at  the 
direction  of  the  Air  Force  Acquisition  Executive.  This  included  the  aggressive  use  of 
CAIV  and  TSPR  and,  as  already  noted,  the  abandonment  of  traditional  military 
specifications  and  standards,  such  as  missile  reliability  requirements.  The  contractor  was 
also  put  in  charge  of  all  testing  and  given  extensive  authority  to  make  trade-offs  within 
nearly  all  aspects  of  the  program  in  order  to  achieve  highly  aggressive  CAIV  objectives. 
A  very  lean  PMO  and  relaxation  of  reporting  requirements  reduced  government  oversight 
to  a  minimum.  Eventually,  these  initiatives  proved  highly  problematic  for  the  program.  A 
very  similar  story  unfolded  for  SBIRS,  again  detailed  in  an  appendix,  which  also  gives 
the  full  story  on  the  use  of  TSPR. 

Evolutionary  Acquisition 

Another  area  of  poor  policy  implementation  is  Evolutionary  Acquisition  (EA).  A 
“proper”  EA  strategy  is  perhaps  best  typified  by  the  E-16  fighter  program,  wherein 
successively  more  capable  “blocks”  of  aircraft  were  acquired  in  successive  budget  years, 
rather  than  seeking  to  build  the  “ultimate”  fighter  in  the  first  tranche. 


^2  Some  have  observed  that  the  reduetion  in  government  oversight  and  management  personnel  paralleled 
a  general  reduetion  in  overall  defense  aequisition  funding  after  the  Cold  War.  This  study  did  not 
attempt  to  partition  sueh  eauses. 
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Although  EA  has  been  DoD  poliey  for  a  number  of  years,  there  are  few  examples 
of  its  sueeessful  implementation  (there  were  none  among  our  MDAP  sample).  Examples 
of  poor  implementation,  on  the  other  hand,  abound.  The  2003  version  of  DoDD  5000.2 
defined  two  versions  of  EA — Ineremental  Development,  under  whieh  requirements  do 
not  ehange,  and  Spiral  Development,  under  whieh  eaeh  “spiral”  ean  have  different 
(usually  inereased)  requirements.  The  2008  reissue  of  5000.2  removed  this  distinetion. 

A  number  of  our  programs  elaimed  to  have  utilized  evolutionary  aequisition 
strategies,  but  few  appeared  to  have  followed  the  evolutionary  model  as  usually 
understood,  or  to  have  aehieved  the  intended  results.  The  basie  prineiple  is  that 
development  will  proeeed  in  a  series  of  diserete  inerements,  with  eaeh  inerement 
involving  worthwhile  performanee  enhaneements  of  limited  seope  ehange  and  little 
risk.53 

The  Spiral  Development  version  has  eaused  the  most  problems.  Of  the  programs 
we  examined.  Global  Hawk  is  the  most  notable  example.  Linder  the  rubrie  of  Spiral 
Development,  the  program  expanded  requirements  in  an  uneontrolled  way  and  permitted 
exeessive  eoneurrency  in  the  needed  development,  testing,  and  low-rate  produetion.  ECS 
formally  invoked  evolutionary  aequisition,  but  the  program  has  not  effeetively  used  its 
evolutionary  strategy  to  avoid  risks  or  eontrol  eosts.  ECS  is  pursuing  a  modified  EA 
approaeh,  in  that  as  the  eomponents  of  ECS  reaeh  maturity  and  readiness  for  produetion 
they  are  being  “spun  out”  for  applieation  within  the  Army’s  eombat  brigades.  This 
strategy,  whieh  was  not  the  original  eoneept  at  MS  B,  seems  to  have  a  reasonably  good 
prognosis  for  sueeess,  if  the  teehnologies  are  shown  to  be  suffieiently  mature  to  warrant 
produetion. 

Defective  Contracting 

The  eontraetor’s  bid  for  JASSM,  whieh  served  as  the  basis  for  the  PM’s  eost 
estimate,  offered  firm  fixed-priees  for  produetion  of  the  first  five  lots,  and  provided  for 
priee  inereases  thereafter.  As  pointed  out  in  the  CAIG  report  at  MS  II,  the  offer  was  on  its 
faee  unsound:  the  eost  of  produetion  for  the  first  five  lots  was  eertain  to  be  higher  than 
the  bid  priee.  The  first  five  lots  were  based  on  a  low  option  priee  offer  earrying  a 
reasonably  high  risk  that  the  government  would  not  be  able  to  exereise  them,  whieh  in 
faet  is  what  happened.  The  result  was  redueed  quantities  and  proeurement  delays  due  to 


^3  For  a  fuller  and  more  authoritative  deseription  of  the  poliey  see  DoD  Instruetion  5000.02  of  Dee  8, 
2008,  p.  13. 


35 


reliability  problems,  eventually  resulting  in  the  loss  of  the  low  option  priees.  The  JASSM 
story  is  thus  an  amalgam  of  mandated  eosting,  poor  management,  and  overly  zealous  use 
of  aequisition  reform  initiatives  that  gave  exeessive  authority  to  the  eontraetor. 

c.  Poor  Contractor  Selection,  Oversight,  and  Incentivization 

We  found  oases  in  whioh  the  eontraetor’ s  management  and  technioal  oapabilities 
proved  to  be  signifioantly  defioient,  where  the  program  offioe  and/or  DCMA  oversight 
was  inadequate  to  identify  and  correot  problems,  and/or  the  eontraetor  inoentives  were 
not  well  aligned  with  the  government’s  interests.  Some  of  these  problems  seem  to  be 
assooiated  with  the  misguided  aequisition  reform  efforts  discussed  above— relating  to 
efforts  to  reduce  burdens  on  contractors  and  empower  them  to  take  more  responsibility 
and  control.  Unfortunately,  this  is  not  always  a  realistic  policy,  and  the  acquisition  system 
has  not  been  consistently  effective  in  identifying  the  extent  to  which  it  can  be  beneficial 
in  particular  cases. 

In  the  LPD-17,  Global  Hawk,  ARH,  and  SBIRS  programs  the  contractors  selected 
were  not  fully  ready  to  do  what  the  program  required  because  personnel  with  appropriate 
skills  and  expertise  were  not  assigned  to  the  program  and/or  critical  tools  proved 
inadequate.  In  all  of  these  cases,  the  government’s  range  of  choice  was  limited  by  a 
paucity  of  bids.  In  the  LPD-17  and  SBIRS,  competition  was  inherently  limited  by  an 
industry  that  had  consolidated  to  the  point  where  very  few  sources  remained.  In  the  case 
of  the  ARH,  the  failure  of  several  potential  competitors  to  bid  was  attributed  to  unduly 
restrictive  KPPs  (a  requirements  issue)  and/or  unrealistic  cost  objectives. Regardless  of 
the  cause,  selection  of  an  inadequately-qualified  (SBIRS)  or  ill-prepared  (ARH) 
contractor  invariably  resulted  in  a  difficult  and  costly  “catch-up”  period  and  less-than 
satisfactory,  or  in  case  of  ARH,  results  so  untoward  that  the  program  had  to  be  cancelled. 

Poorly  designed  incentives,  misuse  of  risk-sharing  in  development  contracts,  and 
conflicts  of  interest  were  also  issues  with  the  JASSM  contract. 

Many  of  the  programs  we  surveyed  suffered  from  weak  oversight  and  contractors 
support.  Specific  examples  include  JASSM,  SBIRS,  ARH,  LPD-17,  EFV,  and  LCS. 
Program  office  staffs  were  small  and  heavily  weighted  with  non-governmental  support 
contractor  personnel.  The  result  was  that  program  managers  often  did  not  become  aware 


In  a  post-mortem  industry-government  session  after  program  eaneellation,  the  eontraetor  was  asked 
what  he  would  have  done  different.  The  answer:  not  bid. 
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of  problems  promptly  and  had  limited  resources  to  deal  with  them  when  they  did  become 
apparent.  We  found  evidence  that  this  situation  was  exacerbated  by  the  relaxation  in 
contractor  reporting  requirements  due  to  acquisition  reform  efforts. 

More  generally,  these  examples  identify  the  difficulty  that  government  source 
selection  authorities  have  in  accurately  ascertaining  the  reasonableness  of  the  contractors 
cost  estimates  when  evaluating  bids  for  cost-type  contracts.  Similar  uncertainties  underlie 
assessments  of  contractors’  technical  and  management  ability  to  execute  their  proposed 
development  program.  Such  source  selection  problems  were  exacerbated  by  the  loss  of 
government  expertise  that  resulted  from  the  reductions  in  the  government  acquisition 
work  force  during  the  1990s. 

2,  Weaknesses  in  Initial  Program  Definition  and  Costing 

Our  second  major  cause  of  acquisition  cost  growth  seems  obvious:  if  a  program’s 
initial  cost  estimate  is  erroneously  low,  future  cost  growth  is  virtually  assured.  While  it  is 
likely  that  there  are,  as  many  believe,  cultural  reasons  for  why  initial  program  cost 
estimates  tend  to  be  low,  those  reasons  are  difficult  to  document.  Instead,  we  sought  to 
determine  whether,  and  if  so  why,  the  initial  estimates  on  the  programs  we  examined 
were  unrealistically  low. 

Before  going  into  detail,  it  is  useful  to  note  that  underestimating  initial  program 
costs  is  very  common  in  projects  of  all  kinds  throughout  government  and  industry,  a 
conclusion  which  is  beyond  our  scope,  but  substantiated  by  a  large  body  of  literature. 
Management  scientists  and  organizational  psychologists  have  developed  extensive 
literature  on  this  phenomenon,  ascribing  it  to  fundamental  cognitive  distortions  in  the 
thinking  of  humans  generally  and  the  political  factors  they  engender.  Rational  choice 
economic  theorists  acknowledge  the  phenomenon  of  widespread  optimism  in  estimates 
while  seeking  to  locate  it  as  a  rational  response  to  underlying  incentives. Although  we 


These  conclusions  come  from  our  interviews.  It  was  beyond  our  scope  to  delve  into  the  details  of  the 
contractual  relationships  and  government  oversight  of  the  contracts  for  the  programs  examined.  Thus 
we  cannot,  in  general,  draw  firm  conclusions  regarding  how  “blame”  for  bad  outcomes  should  be 
assigned  between  the  contractor  and  the  government. 

See  for  example,  Flyvbjerg,  Holm,  and  Buhl,  Underestimating  Costs  in  Public  Works  Projects,  Journal 
of  the  American  Planning  Association,  Summer  2002. 

This  literature  is  far  too  voluminous  to  cite  even  in  summary  here.  Recent  papers  which  contain 
references  to  earlier  studies  include  Jean-Pierre  Benoit,  Juan  Dubra,  and  Don  Moore,  “Does  the 
Better-Than- Average  Effect  Show  That  People  Are  Overconfident?:  An  Experiment,”  MPRA  Paper 
13168  (Munich  Personal  RePEc  Archive,  Feb  2009)  http :// mpra.ub .uni-muenchen. de/ 13168/;  Markus 
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focus  on  more  specific  causes,  it  should  be  remembered  that  these  underlying  factors  are 
at  work  and  should  not  be  neglected.  Our  policy  recommendations  are  consistent  with 
those  advocated  by  experts  in  this  area  of  study. 

Problems  with  initial  cost  estimates  begin  with  the  derivation  of  the  SCP  and  the 
ICE  (after  CAIG  review),  as  described  in  the  Introduction.  For  most,  if  not  all,  of  the 
programs  we  examined  in  detail  it  was  evident  in  retrospect  that  the  initial  MS  B  cost 
estimates  were  unrealistically  low  (that  is,  even  before  considering  subsequent 
unforeseen  events). Our  review  found  no  single  dominant  cause  for  this  phenomenon. 

CAIG  Estimates  and  Cost  Growth 

The  programs  we  studied  reveal  that  funding  to  the  CAIG  estimate  would  not 
have  eliminated  reportable  cost  growth.  Despite  better  overall  performance  when 
compared  to  the  SCPs,  the  CAIG’s  estimates  were  sometimes  significantly  in  error.^^ 
When  this  occurs,  the  entire  decision-making  process  is  put  at  risk,  including  both  the 
original  cost-effectiveness  rationale  and  the  subsequent  stability  and  executability  of  the 
program. 

It  is  not  possible  to  produce  a  realistic  cost  estimate  for  a  poorly  defined  program. 
If  there  is  one  prevailing  reason  for  low  initial  cost  estimates,  both  Service  and  CAIG,  it 
is  a  lack  of  accurate,  sufficiently  detailed,  and  complete  information  about  the  program. 


K.  Brunnermeier,  Filippos  Papakonstantinou,  and  Jonathan  A.  Parker,  “An  Economic  Model  of  the 
Planning  Fallacy,”  NBER  Working  Paper  No.  14228  (National  Bureau  of  Economic  Research,  August 
2008)  http://www.nber.org/papers/wl4228;  Bent  Flyvbjerg,  Massimo  Garbuio,  and  Dan  Lovallo, 
“Delusion  and  Deception  in  larger  Infrastructure  Projects:  Two  Models  for  Explaining  and  Preventing 
Executive  Disaster,”  California  Management  Review  51,  No.  2  (Winter  2009):  170-93;  and  Dan 
Lovallo  and  Daniel  Kahneman,  “Delusions  of  Success:  How  Optimism  Undermines  Executives’ 
Decisions,”  Harvard  Business  Review  81,  No.  7  (July  2003):  29-36.  Earlier  papers  of  fundamental 
importance  include  Roger  Buehler,  Dale  Griffin,  and  Michael  Ross,  “Exploring  the  ‘Planning  Fallacy’: 
Why  People  Underestimate  Their  Task  Completion  Times,”  Journal  of  Personality  and  Social 
Psychology  67,  No.  3  (Sep  1994):  366-81;  Roger  Buehler  and  Daniel  Kahneman,  “Timid  Choices  and 
Bold  Forecasts:  A  Cognitive  Perspective  on  Risk  Taking,”  Management  Science  39,  No.  1  (Jan  1993): 
17-31;  James  G.  March  and  Zur  Shapira,  “Managerial  Perspectives  on  Risk  and  Risk  Taking,” 
Management  Science  33,  No.  11  (Nov  1987):  1404-18;  Amos  Tversky  and  Daniel  Kahneman, 
“Judgment  under  Uncertainty:  Heuristics  and  Biases,”  5'c/ence  185,  No.  4157  (27  Sep  1974):  1124-31. 

This  is  not  surprising,  since  our  selection  of  the  1 1  programs  was  not  a  random  sampling;  rather,  we 
deliberately  chose  high-visibility  programs  that  had  experienced  substantial  cost  growth,  as  measured 
from  the  Milestone  B  estimate. 

This  finding  is  consistent  with  previous  IDA  research,  as  reported  in  McNicol  et  al.,  “The  Accuracy  of 
Independent  Estimates,”  IDA  Paper  P-3989.  Some  of  the  programs  included  in  this  report’s  sample 
have  suffered  significant  increased  cost  growth  since  its  data  cutoff  date  and  this  will  tend  to  increase 
the  degree  to  which  the  ICEs  and  SCPs  fall  short  of  reality. 
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to  permit  a  reliable  eost  estimate  to  be  developed.  This  problem  is  closely  associated  with 
the  immaturity  of  the  system  design,  the  critical  technologies,  or  both.  At  the  time  of  the 
estimate,  even  the  sponsor  may  have  an  inadequate  or  erroneous  understanding  of  how 
the  actual  program  is  likely  to  unfold.  And  as  discussed  earlier,  when  the  initial  stages  of 
program  development,  as  specified  in  DoDD  5000.02,  are  bypassed,  it  is  less  likely  that 
the  program  will  be  fully  defined  and  critical  technologies  matured  at  MS  B,  and  thus  that 
the  initial  cost  estimate  will  be  reliable. 

Similar  errors  occur  when  highly  optimistic  assumptions  about  the  cost-saving 
benefits  of  acquisition  reform  initiatives  are  made  in  the  SCP,  without  direct  supporting 
evidence. 

In  the  case  of  the  Joint  Strike  Fighter  (JSF)  program,  lack  of  adequate  program 
definition  led  the  CAIG  to  underestimate  costs.  The  ICE  was  based  on  early  assumptions 
about  airframe  weights,  materials  content,  the  amount  of  engineering  effort  needed  to 
develop  a  viable  design,  and  how  much  commonality  could  be  achieved  across  disparate 
variants.  The  CAIG's  information  on  these  matters  came  largely  from  the  JSF  program 
office,  and  it  lacked,  for  the  most  part,  any  means  to  verify  the  information 
independently.  Thus,  although  its  procedure  for  estimating  costs  was  independent,  it  did 
not  have  sufficient  independent  engineering  information  when  producing  the  estimate. 
APUC  is  currently  estimated  at  42%  over  the  CAIG  estimate  at  MS  B,  and  most  of  that 
growth  can  be  attributed  to  lack  of  sound  engineering  information  on  which  the  CAIG’s 
cost  estimates  could  be  based.  Other  cases  where  the  program  costed  by  the  CAIG 
differed  considerable  from  what  was  actually  acquired  include  the  ECS,  Global  Hawk, 
ECS,  and  to  a  lesser  extent,  ARH. 

Much  of  the  ECS  program  coming  into  MS  B  was  conceptual  and  realistic  costing 
was  simply  not  possible.^*’  Eor  example,  there  was  virtually  no  specificity  with  regard  to 
the  intended  new  family  of  lighter-weight  armored  vehicles  (the  initial  weight  objectives 
proved  highly  unrealistic)  and  the  broadband  communications  network  envisioned  was 
just  that — a  vision.  The  CAIG  initial  RDT&E  cost  estimate  was  34%  higher  than  the 
SCP,  while  the  procurement  cost  estimate  was  15%  higher.  (The  latest  estimate  before 
cancellation  of  a  large  part  of  the  program  indicated  cost  growth  of  50%  in  RDT&E  and 


The  CAIG  report  at  MS  B  for  PCS,  whieh  is  referenced  extensively  in  the  PCS  annex,  makes  this  point 
in  no  uncertain  terms. 
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66%  in  procurement,  but  those  comparisons  are  confounded  by  numerous  changes  in 
scope,  both  inereases  and  decreases,  over  the  eourse  of  the  program.) 

SBIRS  presents  a  number  of  serious  program  pathologies  that  have  eontributed  to 
more  than  a  250%  unit  cost  growth,  even  with  reduced  scope  and  content.  At  program 
initiation,  the  estimated  cost,  which  was  in  the  $10  billion  range,  was  considered 
unaffordable  in  the  context  of  other  Air  Force  priorities,  prompting  an  intensive  effort  to 
find  less  costly  approaches — without  saerificing  desired  performance.  One  of  the  major 
proposals  to  reduce  costs  was  to  introduce  “commercial  practices”  in  the  management 
and  execution  of  the  program.  It  was  also  expected  that  use  of  commercial  components, 
together  with  a  predicted  large  increase  in  demand  for  commercial  space  systems,  would 
bring  great  economies  of  scale  and  overhead  dilution.  Partieipants  told  us  that  savings  of 
20%  or  more  were  envisioned  as  a  result,  and  these  rosy  projections  found  their  way  into 
the  SCP  and  CAIG  estimates.  Scarcely  any  of  those  optimistic  visions  turned  out  to  be 
accurate.  The  commercial  market  fell  far  short  of  projection  and,  as  documented  in  the 
appendix  on  SBIRS,  “streamlined”  management  contributed  directly  to  program 
problems,  imposing  costs  rather  than  savings. 

Costing  to  Available  Funds 

An  especially  damaging  practice  for  some  SCPs  has  been  costing  programs  on  the 
basis  of  what  will  fit  within  projeeted  program  resources,  rather  than  on  the  objective  cost 
implications  of  the  program  plan  and  teehnical  content. 

The  worst  of  our  1 1  examples  is  the  LCS.  The  Service  chief,  intending  to  support 
a  building  program  eonsistent  with  the  Navy’s  fleet  sizing  goal  and  likely  budgets, 
dictated  the  system’s  unit  cost,  apparently  without  due  consideration  of  the  impracticality 
of  meeting  that  cost  for  an  operationally  viable  warship  with  the  desired  eapabilities.  The 
cost  cap  imposed  by  the  Chief  of  Naval  Operations  (CNO)  was  set  at  less  than  half  of  the 
original  estimate  by  the  Navy’s  ship  cost  group  and  was  nominally  based,  as  deseribed  to 
the  CAIG,  on  a  largely  commercial  vessel,  essentially  a  coastal  fast  ferry  with  crew 
accommodations,  carrying  weapons  modules  in  place  of  vehieles.  The  original  SCP  and 
CAIG  estimates  indeed  turned  out  to  be  less  than  half  of  the  currently  estimated  cost 
because  the  Navy’s  funetional  requirements  for  the  ship  to  deploy  to  distant  theaters  and 
operate  largely  independent  of  local  support,  survive  moderate  levels  of  weapon  damage, 
and  integrate  its  modular  weapons  and  sensors  could  not  be  met  within  the  mandated  cost 
or  size  constraints,  or  on  the  desired  schedule.  In  essence,  while  the  Navy  said  initially 
that  it  wanted  a  ferryboat,  in  the  end  it  insisted  on  getting  a  warship,  and  found  that  it 
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could  not  get  one  for  a  ferryboat  price.  The  industry,  however,  took  the  Navy’s  original 
statements  and  cost  targets  at  face  value  and  submitted  bids  based  on  commercial  ferry 
construction  practice.  Sinee  their  estimate  was  not  supporting  an  MS  B  deeision,  the 
CAIG  did  not  prepare  a  formal  ICE  and  simply  accepted  the  industry  bid  numbers  as 
generally  reasonable,  providing  no  formal  warning  of  the  large  cost  growth  to  come. 

The  story  is  somewhat  similar  for  the  now-caneelled  ARH.  There  was  funding 
available  from  caneellation  of  the  RAH-66  Comanche  program,  so  the  Army  deeided  to 
invest  in  a  new  helicopter  based  on  an  off-the-shelf  commercial  product.  The  intended 
unit  cost  of  the  TRIP  aireraft  was  actually  specified  in  the  Request  for  Proposal  (REP)  - 
in  essence  a  “design  to  price”  aequisition  strategy.  The  winning  design  was  based  on  the 
Bell  417,  a  planned  eommereial  upgrade  of  the  Bell  407  helicopter.  But  the  417  did  not 
go  into  production,  so  all  the  cost  estimates  based  on  assuming  it  would  beeame  invalid. 
The  MS  B  CAIG  cost  estimate  accepted  the  Army’s  assumption  that  the  ARH  would  be  a 
modified  version  of  the  yet-to-be-built  eommereial  417  and  was  reasonably  close  to  the 
SCP,  giving  no  warning  that  the  costing  actually  contained  a  large  element  of  risk.  This 
was  not  so  mueh  a  quantitative  error  in  initial  costing  as  much  as  a  failure  to  identify  a 
signifieant  risk  implieit  in  that  estimate. 

Several  other  programs,  notably  EEV  and  SIBRS,  suffered  from  the  “cost- 
according-to-available  funding”  syndrome,  usually  also  associated  with  extreme 
optimism  regarding  achievable  eeonomies  or  shortcuts.  Eor  those  programs  the  actual 
funding  was  set  at  levels  substantially  below  what  was  realistic  or  what  was  necessary  to 
carry  out  the  planned  development  effort,  foreing  deferral  or  shortehanging  of  some 
tasks.  Bidders  responding  to  REPs  are  almost  invariably  aware  of  the  budgetary 
limitations  under  which  the  program  is  being  initiated  and  are  thus  under  pressure  to  bid 
in  line  with  Service  cost  expectations. 

It  could  be  argued — though  the  Nunn-MeCurdy  Act  says  otherwise — that 
inaccuracies  in  initial  cost  estimates  should  not  be  “scored”  as  real  cost  growth,  but 
simply  as  erroneous  eosting:  that  so  long  as  DoD  gets  value  for  the  money,  it  is  not  so 
important  whether  the  initial  estimate  was  right  or  wrong. Although  such  an  outcome  is 


JASSM  is  an  example  among  the  programs  examined;  despite  the  program’s  eost  overrun  and 
reliability  diffieulties,  the  missiles  have  been  deployed  and  provide  a  eapability  that  eould  be  very 
valuable  in  eertain  high-threat  seenarios. 
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certainly  preferable  to  DoD  spending  a  lot  of  money  on  systems  that  turn  out  poorly, ^2  as 
noted  earlier  there  are  many  adverse  ramifieations: 

•  The  original  cost-effeetiveness  assessment  of  the  new  system  is  ealled  into 
question; 

•  There  is  a  disruptive  impact  throughout  the  program/budget  system 
(partieularly  on  other  aequisition  programs); 

•  The  eredibility  of  DoD  with  the  Congress  and  the  publie  is  diminished;  and 

•  Exeessive  management  attention  is  needed  to  address  cost  growth  issues — re¬ 
baselining,  Nunn-MeCurdy  breaehes,  ete. 

When  the  true  eosts  of  aequiring  a  system  that  is  seriously  exeeeding  its  initial 
eost  estimates  beeome  manifest  (and  that  manifestation  seldom  oeeurs  all  at  onee — it 
beeomes  apparent  only  over  an  extended  period),  several  outeomes  ean  oeeur: 

1.  The  program  may  be  eaneelled  (e.g.,  ARH),  so  the  government  has  wasted 
mueh  if  not  all  the  money  put  into  it  before  eaneellation,  and  the  user  has  to 
do  without  an  important  eapability  until  a  new  program  ean  be  exeeuted;  Sueh 
eaneellations  are  very  rare; 

2.  The  higher  eosts  are  determined  to  be  worth  meeting  without  undue  delay, 
resulting  in  a  squeeze  on  other  programs  to  pay  for  the  overage;  or, 

3.  What  most  frequently  happens,  the  program  is  stretehed  out  to  fit  within 
available  resources — almost  always  with  the  neeessity  of  at  least  some 
additional  funding  to  eover  the  longer  period,  unless  planned  quantities  are 
signifieantly  redueed. 

The  eost  ineffioieney  in  the  latter  two  oases  has  been  estimated  to  be  at  least  2% 
to  8%  of  the  entire  aequisition  budget — some  $2.6  billion  to  $10.4  billion  annually  at 
ourrent  levels,  resulting  from  extensions  of  RDT&E  oontraots  and/or  lower  produotion 
rates  over  a  longer  period.^^ 

a.  Defective  and  Unstable  Requirements  Processes 

Establishing  clear,  affordable,  and  oost-effeotive  requirements  has  been  a 
longstanding  problem  for  the  DoD  aequisition  prooess  (and  indeed  for  all  DoD  resouroe 
allooation  prooesses).  It  is  beyond  the  seope  of  this  study  to  discuss  in  detail  all  of  the 


62  Program  cancellation  may  be  the  best  of  poor  outcomes;  worse  still  is  to  field  systems  that  aren’t 
capable,  or  are  only  marginally  capable  of  performing  the  mission,  or  with  costs  to  operate  and  support 
that  exceed  their  value. 

McNicol,  “Cost  Growth  in  Major  Weapon  Procurement  Programs,”  IDA  Paper  P-3832,  pp.  79-87.  The 
author  warns  that  the  cost  might  well  be  even  greater  than  8%. 
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problems  with  the  aequisition  requirements  process;  however,  deficiencies  in  the  process 
were  key  factors  in  cost  growth  for  several  of  the  programs  we  studied.  In  those  cases, 
requirements  appear  to  have  been  established  without  adequate  evaluation  of  their  risk 
and  cost  implications,  imposed  by  top-level  fiat  without  adequate  analysis  of  real  need  or 
cost-effectiveness,  or  changed  radically  in  the  course  of  the  program  without  a  full 
understanding  of  the  impacts. 

For  most  of  the  programs  in  our  sample,  there  was  little  or  no  evidence  of  early 
studies  examining  tradeoffs  of  system-level  requirements  against  cost  (or,  equivalently, 
against  the  quantity  that  can  be  afforded  at  a  given  budget  level)  or  risk,  although  some 
tradeoffs  became  necessary  as  cost  or  technological  realities  undermined  the  original 
program  plans.  In  several  cases,  risks  or  added  scope  associated  with  poorly  analyzed 
requirements  have  been  major  drivers  of  cost  growth. 

One  clear  illustration  is  Global  Hawk.  The  Air  Force  initially  opposed  OSD’s 
initiative  to  start  a  program  of  record  based  on  the  DARPA  ACTD.  Once  it  was  forced  to 
begin,  however,  the  Air  Force  embraced  a  much  more  ambitious  version  than  OSD  had 
envisioned  and  approved.  MS  II  approval  to  enter  into  SDD  was  based  on  the  DARPA 
ACTD  program  that  included  only  one  major  sensor  suite  per  aircraft.  Almost 
immediately  after  the  milestone  decision,  however,  the  Air  Force  initiated  a  study  of 
sensor  requirements  that  resulted  in  a  50%  increase  in  the  payload  required  of  the  air 
vehicle.  Gross  underestimation  by  the  contractor  and  the  government  of  the  changes  in 
airframe  design  that  would  be  required  led  to  a  spiral  of  cost  increases — increases  that 
were  further  exacerbated  by  other  factors  discussed  elsewhere  in  this  report.  Given  such  a 
major  increase  in  system  requirements,  the  appropriate  course  may  have  been  to  proceed 
with  more  production  and  fielding  of  the  “A”  model,  a  smaller  and  simpler  air  vehicle 
based  closely  on  the  ACTD  design,  as  planned  at  MS  II,  until  the  requirements  for  a 
larger  vehicle  could  be  more  fully  defined  and  its  design  and  cost  impacts  thoroughly 
understood.  If  the  larger  vehicle  could,  in  fact,  be  justified,  a  new  program  could  have 
been  initiated  as  it  had  for  Predator  B  (another  remotely  manned  ISR  aircraft). 

We  have  already  mentioned  the  case  of  the  LCS,  where  lack  of  clarity  about 
requirements  led  to  confusion  about  the  basic  nature  of  the  ship,  and  as  a  result  the 
doubling  of  both  cost  and  the  duration  of  development.  In  the  ARH  program  a 
requirement  that  two  helicopters  fit  within  a  C- 130’s  cargo  compartment,  sharply 
narrowing  the  potential  field  of  competition,  was  adopted,  apparently  without  formal 
consideration  of  the  implications  or  possible  alternatives. 
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In  the  JASSM  missile  program,  in  the  name  of  aequisition  reform,  traditional 
military  speeifications  and  standards,  sueh  as  missile  reliability  requirements,  were 
abandoned.  Instead  of  speeifying  a  stand-alone  reliability  requirement,  the  JASSM 
program  relied  on  a  broader  and  unproven  approaeh  to  performance-based  requirements 
that  relied  heavily  on  modeling  and  simulation  to  determine  “Missile  Mission 
Effectiveness”  (MME)  and  the  “probability  of  reliability  growth.”  This  approach  assessed 
only  the  product  of  three  parameters,  lethality,  survivability,  and  reliability;  although 
there  was  no  lower  bound  or  minimum  requirement  for  any  one  parameter.  The  missile 
excelled  at  the  first  two  parameters,  but  has  (so  far)  failed  to  achieve  the  level  of 
reliability  considered  acceptable  to  the  operators.  Although  the  contractor  met  the 
program’s  effectiveness  objective  as  defined  by  the  “MME,”  DOT&E  subsequently 
deemed  the  missile’s  reliability  as  inadequate.  Thus,  a  reliability  requirement  was  added 
retroactively  to  the  program.  The  entire  episode  contributed  very  substantially  to  cost 
growth  in  both  the  RDT&E  and  procurement  costs. 

JTRS-GMR  originally  had  a  fairly  simple  and  clearly  defined  requirement  to 
develop  a  low-cost  radio  system  with  software-defined  waveforms  to  replace  a  variety  of 
existing  tactical  radios,  using  commercial  technology.  Before  MS  B,  however,  the 
requirement  was  revised  to  include  development  of  an  ad-hoc  battlefield  tactical  network 
-  essentially  a  secure  and  reliable  broadband  cellular  communications  system  on  the 
battlefield,  without  cell  towers.  This  has  led  to  substantially  increased  costs  and  schedule 
slippage,  and  its  feasibility  remains  in  doubt,  but  it  was  adopted  without  apparent  serious 
review  of  the  consequences.  Moreover,  the  interoperability  benefits  originally  envisioned 
for  the  program  have  not  been  obtained. 

Notwithstanding  the  evidence  presented  above,  it  is  important  to  acknowledge 
that  some  requirements  changes  can  be  shown  to  be  justified.  Eor  example,  the  emerging 
availability  of  new  technology  may  well  permit  the  fielding  of  a  much  more  effective  or 
reliable  system  at  only  modest  increases  in  development  cost.  Einancial  reserves  for  such 
eventualities  would  be  a  significant  aid  to  program  stability. 

b.  Entry  into  Development  with  Immature  Technologies 

Another  well-settled  basic  principle  of  sound  acquisition  planning  is  to  ensure  that 
all  technologies  critical  to  the  eventual  performance  of  the  intended  system  have  reached 
an  adequate  level  of  maturity  before  the  full  project  is  approved  for  entry  into  full-scale 
engineering  and  manufacturing  development.  However,  few  of  the  programs  we 
examined  came  close  to  meeting  that  standard.  Only  a  small  fraction  of  the  key 
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technologies  were  sufficiently  mature,  yet  the  PCS,  SBIRS,  JTRS-GMR,  and  WIN-T 
programs  were  approved  to  proceed  into  development.  In  some  cases  this  may  have  been 
to  avoid  the  appearance  that  the  Department  was  losing  momentum  on  initiatives  that  had 
high  level  support.  Even  if  individual  technologies  are  properly  assessed  as  ready  to 
support  full-scale  development,  integration  is  critical  at  the  system  of  systems  level,  and 
currently  there  is  no  assessment  made  of  the  readiness  of  technologies  to  achieve  that 
integration.  This  issue  is  closely  related  to  shortfalls  in  front-end  systems  engineering, 
discussed  in  the  next  sub-section. 

Despite  the  requirements  of  DoDI  5000.2,  most  of  the  systems  in  our  sample 
failed  to  demonstrate  the  required  technology  readiness  levels  (TRLs)  for  all  their 
technologies  prior  to  MS  B,  and  some  did  so  egregiously.  ECS,  JTRS-GMR,  and  WINT 
are  the  most  extreme  examples  of  the  problem.  In  a  May  2003  memorandum  reporting  on 
a  review  of  the  Army’s  Technology  Readiness  Assessment  for  ECS,  the  Director, 
Defense  Research  and  Engineering  stated  that  24  of  the  3 1  identified  critical  technologies 
were  at  TRLs  below  6,  but  MS  B  was  approved  anyway.  In  Eebruary  2009 — six  years 
later — the  Army  stated  that  four  critical  technologies  were  still  at  TRLs  below  6. 
Eurthermore,  those  four  technologies  were  at  the  heart  of  the  “mobile  ad  hoc  network” 
needed  to  achieve  the  capabilities  on  which  ECS  is  premised.  JTRS-GMR  successfully 
passed  MS  B  with  none  of  its  20  critical  technologies  at  the  TRL  6,  and  some  below  TRL 
4.  And  WIN-T  had  only  3  of  12  critical  technologies  at  the  TRL  6. 

The  ARH  was  another,  somewhat  different,  example.  Since  the  acquisition 
strategy  envisioned  a  modified  commercial  off-the-shelf  (COTS)  helicopter,  the  Army 
was  permitted  to  go  directly  to  MS  B.  There  had  been  no  Concept  Decision  (CD)  or 
Materiel  Development  Decision  (MDD)  or  MS  A  review,  and  no  systematic  Materiel 
Solutions  Analysis  (MSA)  or  Technology  Development  (TD)  processes,  or  their 
equivalents.  After  MS  B,  the  Army  decided  to  substitute  the  Target  Acquisition  Sensor 
System  (TASS)  for  the  sensor  system  originally  selected  by  Bell  Helicopter,  even  though 
there  were  technology  readiness  issues  with  TASS.  In  subsequent  user  testing,  the  system 
experienced  a  major  failure. 

c.  Deficient  Front-End  Analysis  of  System-Level  Design  Issues  and 
Technical  Risks 

Many  of  the  programs  we  examined  showed  evidence  of  inadequacies  in  initial 
systems  design,  systems  engineering,  and  risk  assessment.  Those  inadequacies  translate 
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directly  into  unreliable  initial  cost  estimates  because  they  are  a  reflection  of  poor  program 
definition. 

Systems  Engineering  Shortfalls 

A  common  theme  in  our  case  studies  was  the  notable  lack  of  systems  engineering 
discipline  before  these  complex  programs  entered  full-scale  development.  Without  the 
comprehensive  allocation  of  work  effort  that  results  from  sound  systems  engineering, 
realistic  scheduling  and  material  and  labor  cost  estimating  was  essentially  impossible. 

The  deficiencies  in  early  systems  engineering  we  observed  resulted  in  part  from 
the  effort  to  sharply  limit  spending  in  the  early  years  of  program  development.  Systems 
engineers  are  expensive  and  scarce  and,  under  historical  policies  and  practices,  not 
considered  highly  essential  in  the  early  phases  of  program  definition.  These  shortfalls  in 
early  systems  engineering  shortchanged  MDAPs  both  at  the  program-office/commodity- 
command  level  and  at  the  OSD  review  level,  where  there  has  been  a  dearth  of 
experienced  systems  designers  capable  of  providing  and  supporting  the  type  of 
independent  cost  review  provided  by  the  CAIG. 

The  results  of  these  deficiencies  in  early  systems  engineering  included  serious 
underestimation  of  the  scope  of  the  required  development  effort,  a  failure  to  identify  key 
risk  areas  that  warrant  extra  effort,  and  requirements-design  mismatches.  These  factors, 
individually  and  collectively,  invariably  led  to  high  cost  growth  relative  to  initial 
estimates,  at  least  for  development  and  probably  for  production  as  well.  This  is  a  problem 
that  extends  far  beyond  the  commonly-cited  defects  in  front-end  systems  engineering 
processes.  Almost  without  exception,  the  cases  we  examined  have  involved 
underestimation  of  the  systemic  risks  related  to  the  design  as  a  total  system,  beyond  the 
risks  associated  with  particular  technology  elements.  This  has  contributed  greatly  to  the 
underestimation  of  costs  and  schedules  and,  in  several  cases,  has  led  to  major  and  costly 
development  problems. 

As  observed  earlier,  many  of  these  programs  have  fallen  short  of  TRLs  required 
prior  to  entry  into  full-scale  development.  A  separate  problem  lies  in  inadequate  analysis 
of  design  issues  and  integration  risks  at  the  system  or  system-of-systems  level,  contrasted 
with  individual  technology  areas. 

The  WIN-T  and  JTRS-GMR  are  two  examples.  Both  were  premised  on  an 
assumption  that  communications  architectures  available  or  being  developed  in  the 
commercial  sphere  would  serve  very  different  military  needs,  without  adequate 
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engineering  analysis  to  determine  how  realistie  those  assumptions  were.®"^  In  fact,  both 
programs  have  proven  to  involve  major  unforeseen  challenges,  have  been  substantially 
restructured,  and  have  experienced  large  cost  and  schedule  overruns. 

Another  major  example  is  JSF,  where  delayed  recognition  of  the  overall  design 
and  integration  issues  associated  with  the  unprecedented  supersonic  STOVL  variant  led 
to  a  crisis  halfway  into  SDD  that  threatened  the  viability  of  the  entire  program  and  forced 
design  changes  resulting  in  major  increases  in  development  and  production  costs.  At  the 
time  of  MS  B,  weight  growth  was  forecast  to  be  toward  the  bottom  end  of  the  historical 
range,  based  on  the  use  of  advanced  design  tools.  Only  after  critical  design  review 
(CDR),  when  design  and  construction  of  prototypes  were  well  advanced,  was  it 
recognized  that  the  weight  growth  was,  in  fact,  well  above  historical  norms,  and  at  a  level 
that  would  make  it  impossible  to  meet  operational  requirements.  This  led  to  a  very  costly 
redesign  and  lengthy  delay.  More  thorough  early  systems  engineering  would  have 
identified  the  extent  of  the  weight  risk  much  earlier  and  in  doing  so  avoided  a  great  deal 
of  cost  growth  and  delay  through  more  accurate  budgeting  and  scheduling. 

SBIRS  is  a  particularly  striking  example  of  inadequate  early  systems  engineering. 
Because  the  potential  to  correct  problems  is  so  small  once  satellites  are  launched,  space 
programs  have  traditionally  engaged  in  especially  intensive  systems  engineering  in  their 
early  phases.  The  government  maintained  its  own  high-level  systems  engineering 
capabilities,  both  in-house  and  at  FFRDCs,  in  addition  to  requiring  contractor  systems 
engineering  efforts.  But  SBIRS,  notably,  lacked  systems  engineering  efforts  by  the 
government  or  by  the  contractor,  and  as  a  result  many  very  costly  problems  emerged  long 
after  they  should  have. 

In  the  ARH  program,  as  already  noted  with  regard  to  initial  costing,  there  was  a 
failure  to  recognize  the  substantial  risks  associated  with  selecting  the  Bell  417 
commercial  helicopter  as  the  basic  platform  for  ARH.  The  417  was  in  development  at  the 
time  of  the  contract  award;  however,  it  was  cancelled  because  a  commercial  market 
analysis  failed  to  support  going  into  production.  Despite  that  unexpected  development, 
the  program  continued  to  try  to  execute  an  already  ambitious  schedule.  With  the 
cancellation  of  the  417,  the  planned  production  line,  which  had  been  projected  to  be 
producing  commercial  helicopters  at  a  substantial  rate,  moved  from  Canada  to  a 


According  to  GAO,  at  least  two  studies  were  available  at  the  time  indicating  that  the  assumption  was 
faulty. 
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production  facility  in  Texas  that  had  much  higher  labor  and  overhead  rates  and  was  not 
produeing  new  helicopters  at  the  time.  A  more  deliberate  front-end  analysis  eould  have  at 
least  revealed  that  risk.  Poor  systems  engineering  also  affeeted  the  program.  After  the 
contract  award,  the  Army  deeided  to  use  different  mission  equipment  and  other 
components  than  originally  proposed  by  Bell.  Some  of  those  eomponents  had  teehnology 
readiness  issues  (as  noted  above)  and  some  were  not  well-suited  for  the  ARH  airframe 
design.  These  issues  could  have  been  sorted  out  and  risks  more  clearly  identified  by  a 
more  deliberate  early  systems  design  review. 

Virtually  every  program  we  surveyed  experieneed  eostly  problems  that  could 
have  been  avoided  or  ameliorated  through  better  front-end  analysis  of  overall  design 
issues  and  risks.  Serious  attention  to  system-level  risk  seems  to  have  been  lacking  on  the 
part  of  senior  deeision-makers. 

Inadequate  Recognition  and  Hedging  of  Program  Risks 

In  too  many  cases  major  program  risks  have  not  been  clearly  foreseen  or  hedged 
against.  And  when  risks  have  been  identified,  the  response  has  often  fallen  short.  There 
are  examples  in  praetieally  every  MDAP  we  reviewed,  but  we  will  foeus  on  three. 

Because  spaee  systems  require  very  low  weight,  very  high  levels  of  performance, 
and  very  high  levels  of  reliability,  reeognized  best  practices  call  for  parallel  development 
of  two  or  more  alternative  approaehes  to  critical  components  or  subsystems.  In  many  of 
these  areas,  however,  the  SBIRS  program  followed  a  single-threaded  approach.  The 
result  was  that  several  major  efforts  failed  to  yield  the  neeessary  results,  foreing  program 
stretch-outs  for  replacement  developments  that  resulted  in  major  growth  in  development 
eosts. 

As  noted  earlier,  the  JSF  program  assumed  that  weight  growth  over  the  eourse  of 
EMD  would  be  at  the  low  end  of  the  historieal  range  for  fighter  programs  in  general. 
When  this  proved  to  be  seriously  optimistic,  the  consequences  were  even  more  severe 
than  those  of  excessive  weight  growth  in  normal  aircraft  programs  because  the  STOVL 
variant  was  so  weight-critieal.  Of  eourse,  it  would  have  been  possible  to  antieipate  that 
there  was  a  risk  of  greater-than-hoped  weight  growth  and  that  if  this  oceurred  it  would 
endanger  the  program,  but  warnings  to  this  effect  were  not  acted  upon  or  incorporated  in 
program  planning.  As  a  result,  when  reeognition  of  the  weight  growth  finally  did  come, 
after  the  critieal  design  review,  it  foreed  a  costly  major  reorientation  of  the  development 
program,  and  scrapping  of  much  of  the  previous  work. 
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The  JTRS-GMR  program  proceeded  into  uncharted  waters  by  trying  to  develop  a 
mobile  ad  hoc  communications  network  that  could  function  effectively  in  the 
environment  of  ground  combat  operations  without  recognizing  or  hedging  against  the 
large  technological  risks.  The  result  was  a  five-year  extension  in  the  R&D  program  and 
commensurate  delays  in  being  able  to  produce  the  radios  needed  for  forces  deploying  to 
ongoing  operations.  Instead,  large  numbers  of  legacy  radios  were  procured,  thus 
foregoing  the  interoperability  benefits  the  original  JTRS-GMR  was  supposed  to  achieve. 

In  the  late  1960s  and  1970s,  effective  systems  engineering  technical  oversight  was 
frequently  provided  by  the  predecessors  of  AT&L  (which  had  various  titles  that  changed 
over  time).  The  organization  employed  a  significant  number  of  well-qualified  engineers 
and  the  leadership,  also  technically-oriented,  understood  and  accepted  their  advice  and 
were  more  proactive  in  program  oversight.  That  model  faded  in  the  1980s  as  a  result  of 
increasing  concerns  about  conflicts  of  interest  (which  limited  the  circulation  of  senior 
engineers  between  industry  and  government),  and  a  growing  belief  that  OSD  should 
leave  technical  matters  to  the  Services  and  industry.  But  today,  even  the  Services  lack 
technical  expertise  in  the  government  workforce  — one  of  the  unfortunate  consequences 
of  the  major  reductions  in  the  acquisition  workforce  in  the  late  1990s.  Several  programs 
we  examined  exhibited  these  affects,  namely  PCS,  JASSM,  SIBRS,  and  Global  Hawk, 
and  probably  ARH  as  well.  For  PCS,  the  Army  had  to  turn  to  a  “Lead  Systems 
Integration”  contractor  to  oversee  the  entire  program  and  perform  many  of  the  functions 
normally  performed  by  the  Government. 

d.  Excessive  Schedule  Compression  and  Concurrency 

Most  of  the  programs  examined  had  some  form  of  schedule  compression  and/or 
concurrency  during  at  least  some  phases  of  the  program  that  was  justified  on  the  basis  of 
speeding  the  delivery  of  desired  capabilities  and/or  cutting  costs  by  reducing 
development  and  production  spans.  Unfortunately  all  too  frequently,  these  measures  have 
backfired  and  actually  resulted  in  later  deliveries  and/or  higher  costs  than  might 
otherwise  have  been  the  case.  The  lack  of  robust  early  systems  engineering  contributed  to 
generally  unrealistic  hopes  that  the  development  schedule  could  be  more  compressed 
than  was  the  case  for  previous  successful  developments.  The  result  was  excessive 
concurrency  without  adequate  risk  mitigation  funding  and  inadequate  developmental 
testing,  leading  to  the  need  for  rework  and  retesting. 

Extending  the  development,  program  delays,  or  reduced  quantities  in  production 
always  increase  unit  costs  because  overhead  expenses  continue  for  longer  and  are 

49 


allocated  over  fewer  units.  It  usually  means  more  direet  labor  as  well,  at  least  in 
development,  thus  further  inereasing  costs.  Thus  underestimates  of  required  time  spans 
generally  go  with  underestimates  of  required  resourees,  but  the  damage  ean  be  even 
greater  if  eapabilities  really  needed  by  the  user  are  not  delivered  as  planned. 

A  elear  example  is  the  LCS  program,  where  a  highly  aggressive,  success-oriented 
sehedule  for  the  first  prototype  ship  resulted  in  a  great  deal  of  re-done  and  out-of-scope 
work  when  the  full  needs  of  a  warship  development  program  were  recognized.  The 
resultant  eosts  were  substantially  greater  than  they  eould  have  been  if  the  schedule  had 
been  realistic  to  begin  with.  The  EFV’s  critical  design  review  was  held  immediately 
following  EMD  eontract  award,  before  tests  of  the  teehnology  prototypes  had  been 
completed  or  fully  analyzed.  As  a  result  the  design  of  the  EMD  prototypes  did  not 
incorporate  important  lessons  subsequently  revealed  in  testing,  not  only  resulting  in 
downstream  rework  but  also  eontributing  to  the  ultimate  failure  of  the  EMD. 

ERIE  for  the  larger  variant  of  Global  Hawk  (RB4B)  commenced  before  the 
designs  were  stable  and  long  before  sensor  packages  were  fully  developed  and  ready  for 
integration,  resulting  in  extensive  rework,  not  only  inereasing  eosts  but  further  delaying 
deployment  of  eritical  operational  needs.  In  the  case  of  JASSM,  the  original  development 
program  was  curtailed  (reducing  both  the  length  of  time  that  competition  was  maintained 
and  the  time  available  to  mature  teehnology  and  production  processes),  eontributing  to 
missile  reliability  problems,  rework,  schedule  slippage,  and  cost  growth. 

The  grave  weight  problems  of  the  JSF  were  diseovered  only  after  CDR.  By  then 
mueh  had  been  invested  in  eomponents  for  EMD  prototypes  that  had  to  be  scrapped  or 
extensively  reworked  in  order  to  achieve  flyable  weight.  Moreover,  there  had  already 
been  eonsiderable  work  to  develop  the  details  of  designs  that  had  to  be  discarded  and 
restarted. 

The  initial  systems  engineering  inadequacies  for  ARH  had  a  compounding  effect 
in  that  the  development  sehedule  was  highly  eompressed  beeause  of  a  pereeived  urgent 
need  and  the  assumption  that  the  development  eould  eapitalize  on  a  soon-to-be-available 
commercial  helicopter  and  military  eomponents  from  other  programs.  Even  if  all  the 
going-in  assumptions  had  been  met,  the  sehedule  was  extremely  suceess-oriented.  Then, 
as  previously  noted,  the  Army  directed  major  configuration  changes  after  MS  B  and  Bell 
had  eancelled  the  417. 
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B,  OTHER  OBSERVATIONS 


This  section  discusses  four  problems  areas  for  DoD  acquisition  programs  that, 
while  important  in  their  own  right,  are  more  secondary  causes  of  cost  growth  in  the 
programs  we  examined. 

1.  Lack  of  or  Problematic  Competitive  Prototyping 

Competitive  prototyping  was  a  notable  feature  of  many  aircraft  acquisition 
programs  in  the  two  decades  prior  to  World  War  II. This  practice  was  particularly 
essential  because  aircraft  performance  often  fell  short  of  predictions  in  that  period. 
During  World  War  II  competitive  prototyping  went  into  partial  eclipse,  and  it  was  only 
partly  revived  in  the  years  following  the  war.  With  aircraft  costs  rising  rapidly  and 
performance  predictions  improving,  full-scale  competitive  prototyping  was  used  only 
sparingly  in  the  late  1950s  and  1960s. 

A  series  of  poor  outcomes  in  aircraft  and  other  programs  led  to  a  revival  of 
prototyping  as  a  policy,  under  the  rubric  of  “fly-before-buy,”  early  in  the  1970s.  It  was 
not  without  its  critics,  however,  and  late  in  the  decade  a  task  force  of  the  Defense  Science 
Board  (DSB)  warned  that 

The  widespread  or  mandatory  use  of  full-scale  system  prototypes  for  all 
programs  up  to  the  production  prototype  level  is  frequently  wasteful  of  critical 
national  resources — dollars  and  manpower  as  well  as  time.  There  are  examples  in 
recent  programs  (e.g.,  A-lO/A-9,  F-16/F-17)  where  little  benefit  can  be  found  in 
the  use  of  prototypes  in  terms  of  shortening  the  development  cycle,  reducing 
overruns,  reducing  overall  cost,  or  minimizing  risk.^^ 

It  is  important  to  note  that  the  criticism  is  restricted  to  competitive  “full-scale 
system  prototypes.”  The  use  of  engineering  test  articles  to  prototype  and  test  specific 
features  that  involve  significant  unknowns  or  risks  is  of  undisputed  value.  What  was 
questioned  was  the  need  for  complete  aircraft,  or  their  equivalents  in  other  fields,  which 
tended  to  be  far  more  costly  and  time-consuming  to  build.  Many  systems  engineers  and 
engineering  executives  questioned  the  return  on  such  investments.  The  overall  thrust  of 
the  advice  from  the  DSB  and  other  authoritative  sources  has  been  that  competitive 


“Competitive  prototyping”  must  be  distinguished  from  development  prototyping.  Thorough  testing  of 
full-seale  prototypes  elosely  representative  of  the  final  produetion  artiele  is  a  neeessity  in  the 
development  of  virtually  any  program  involving  substantial  system-level  risk  where  series  produetion 
is  eontemplated.  In  most  eases  eompetitive  prototypes  are  eonstmeted  long  before  the  final  produetion 
eonfiguration  has  been  fully  defined  and  represent  only  aspeets  of  the  final  design. 

Defense  Seienee  Board,  Report  of  the  Acquisition  Cycle  Task  Force,  15  Mareh  1978,  p.  53. 
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prototyping  can  be  a  powerful  and  valuable  tool,  but  that  its  applieation  must  stem  from 
an  analysis  of  the  needs  and  the  cireumstanees  of  particular  programs. 

Competitive  prototypes  between  two  firms  were  planned  through  all  of  Program 
Definition  and  Risk  Reduetion  (PDRR)^^  and  into  the  EMD  phase  of  the  JASSM 
program,  but  those  plans  were  eurtailed  beeause  of  high  eosts  and  Congressional  budget 
euts.  While  the  original  Global  Hawk  aireraft  approved  at  MS  II  (RQ-4A)  was  prototyped 
during  its  ACTD,  the  mueh  larger  RQ-4B  model  that  was  added  to  the  program  within 
months  after  MS  II  was  never  prototyped.  It  was  erroneously  assumed  that  there  would 
be  80%  commonality  between  the  RQ-4A  and  RQ-4B.  Those  assumptions  did  not  prove 
aceurate  and  the  gross  underestimation  of  the  eomplexity  of  moving  to  the  RQ-4B 
resulted  in  signifieant  eost  inereases  and  performance  and  sehedule  shortfalls  in  the 
program. 

Based  on  our  analysis,  the  use  of  eompetitive  prototyping  in  the  JSF  and  LCS 
programs  does  not  appear  to  have  been  eost-effeetive.  In  both  cases  it  is  likely  that  the 
impaet  on  eosts  exeeeded  the  eost  of  the  prototypes  themselves;  thus,  eompetitive 
prototyping  aetually  eontributed  to  their  eost  growth.  (But  for  JFS  those  eosts  were 
ineurred  before  MS  II,  so  any  impaet  on  post  MS  II  eost  growth,  as  reeorded  by  the  study, 
is  purely  eonjeetural.) 

As  previously  discussed,  weight  was  (and  is)  an  espeeially  erucial  issue  with 
regard  to  JSF.  Any  fighter  aireraft  development  involves  a  struggle  with  weight,  but  the 
STOVF  requirement  made  the  aireraft  still  more  sensitive  to  weight,  while  the  three-in- 
one  eommonality  requirement  raised  previously-unexplored  questions  about  aehieving 
the  desired  weight.  The  primaey  of  weight  as  a  system-level  risk  factor  posed  ehallenges 
for  the  pre-MS  II  eompetitive  prototyping  program-ehallenges  that  were  not  met.  To 
aeeurately  replieate  the  weight  of  the  produetion  aireraft  requires  a  prototype  effort 
largely  equivalent  to  that  required  for  the  late  FMD  pre-produetion  prototypes,  whieh  is 
to  say  many  hundreds  of  engineer-years  and  a  great  deal  of  engineering  test  effort, 
requiring  one  billion  dollars  or  more  in  addition  to  the  eost  of  prototype  eonstruetion.^^ 
But  the  opportunity  eosts  to  the  program  were  even  greater.  If  the  same  sum  had  instead 


The  term  used  for  the  pre-MS  B  phase  of  a  program  at  the  time. 

Beeause  the  eontraetors  eontributed  some  of  the  eost  (whieh  beeame  part  of  their  eost  basis  for 
amortization  in  overhead  burden  on  future  eontraets)  there  is  no  elear  aeeounting  of  the  final  aetual 
eost. 
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been  devoted  to  probing  weight  issues  through  early  systems  engineering,  the  weight 
problems  would  likely  have  emerged  much  earlier  in  the  overall  development  process. 
The  result  would  have  been  both  a  better  basis  for  realistic  cost  estimates  at  MS  II  and 
partial  or  complete  avoidance  of  the  crisis  that  later  emerged,  with  all  the  waste  and  delay 
that  it  precipitated. 

The  LCS  program  was  initiated  with  the  construction  of  two  technology 
demonstrator  ships  built  with  RDT&E  funds,  designated  LCS  1  and  LCS  2.  While  these 
ships  were  intended  to  eventually  be  deployable,  they  can  be  considered,  in  effect, 
competitive  prototypes.  Built  in  great  haste,  the  two  initial  units  came  in  far  over  budget 
and  behind  schedule.  It  is  not  at  all  clear  that  the  lessons  learned  in  their  early  testing  will 
translate  into  significantly  less  costly  follow-on  ships,  thus  calling  into  question  their 
value  as  “competitive  prototypes.”  All  of  this  suggests  that  the  more  than  one  billion 
dollars  spent  on  the  two  initial  ships  may  not  provide  a  particularly  good  return  on 
investment. 

2.  Inadequate  Affordability  Assessments 

As  explained  in  the  Introduction,  DoDD  5000.02  requires  that  for  a  program  to  be 
approved  at  MS  B  for  Engineering  and  Manufacturing  Development  the  sponsoring 
Component  must  have  programmed  the  full  funding  for  the  program  in  their  POM.^o 
Once  that  POM  is  reviewed  and  approved  by  the  Secretary  of  Defense,  the  program  will 
be  fully  funded  in  the  DoD  LYDP.  Thus  by  definition,  the  program  is  “affordable”  for  the 
years  covered  by  the  LYDP;  however,  in  most  cases  the  LYDP  reflects,  at  best,  only  early 
procurement  costs  for  new  MDAPs.  When  the  major  costs  occur  beyond  the  LYDP,  as  is 
usually  the  case,  there  is  no  real  mechanism  to  assess  the  extent  to  which  a  proposed 
program  will  be  affordable  in  the  longer  term,  when  other  needs,  overall  DoD  priorities, 
and  likely  fiscal  constraints  are  considered. 

The  Defense  Program  Projection  (DPP),  while  not  an  approved  product  of  the 
DoD  PPBES,  does  provide  a  context  in  which  long-term  affordability  can  be  evaluated, 
although  its  utility  is  limited  by  the  very  fact  that  it  is  not  “approved”  and  therefore  not 


Because  weight  was  so  clearly  a  critical  issue,  this  would  have  been  a  plausible  application  of  added 
funds  if  they  had  been  available. 

The  Component’s  POM  will  subsequently  be  submitted  to  and  reviewed  by  OSD,  which  must  verify 
that  the  program  is  still  fully  funded.  Such  verification  must  occur  with  each  subsequent  POM 
submission  as  well.  If  these  verifications  do  not  occur,  there  is  no  guarantee  that  the  acquisition 
program  is  adequately  funded. 
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authoritative.  Nonetheless,  eonsideration  of  the  outyear  affordability  of  a  proposed 
program  in  the  DPP  eontext  ean  provide  useful  insights  into  its  affordability. 

So,  the  reader  might  ask,  what  does  affordability  have  to  do  with  eost  growth? 
The  answer  is  that  there  is  an  indireet  effeet.  If  a  program  is  initiated  and  later  turns  out 
not  to  be  as  affordable  as  planned,  the  program  will  have  to  either  be  modified  or 
eaneelled.  If  eaneelled,  the  money  already  spent  will  have  been  wasted,  and  the 
requirement  that  the  program  was  supposed  to  fill  will  go  unsatisfied  until  another 
solution  ean  be  found.  If  modified,  there  eould  be  a  reduetion  in  seope  to  make  it 
affordable  or  the  program  can  be  stretched  out  to  require  less  funding  per  year  over  more 
years  (or  both).  Stretch-outs  will  normally  be  accompanied  by  substantial  total  cost 
growth,  both  from  stretching  the  development  program,  as  well  as  lowering  production 
rates,  which  increases  unit  production  costs. 

Even  more  unfortunate  outcomes  can  occur  from  attempts  to  execute  seriously 
underfunded  programs  based  on  a  “can  do”  spirit  and  the  belief  that  shortcuts  and 
“efficiency  measures”  can  achieve  the  improbable.  Such  high  risk  strategies  usually 
backfire,  with  unfortunate  consequences  in  program  execution  and  cost  growth.  Global 
Hawk,  among  the  programs  examined,  was  a  particularly  noteworthy  case.  The  Air  Force 
did  not  have  sufficient  funds  to  properly  execute  the  program,  once  it  had  been  decided  to 
develop  a  larger  aircraft  with  multi-sensor  intelligence  suites,  and  it  was  unwilling  to  take 
resources  from  other  programs.  Instead  it  attempted  to  execute  the  program  with  reduced 
funds,  with  predictably  poor  results. 

ECS,  if  executed  as  defined  at  MS  B,  would  have  entailed  huge  funding 
requirements  in  the  outyears  beyond  the  FYDP  -  requirements  that  are  significant  when 
compared  to  total  Army  investment  program.  It  is  difficult  to  envision  how  those  funding 
demands  could  possibly  have  been  met.  The  only  realistic  expectation  is  that  they  would 
not  have  been,  and  that  ECS  procurement  would  be  long  deferred  and/or  stretched  out. 
Either  option  would  lead  to  substantial  increases  in  costs,  further  exacerbating  this 
program’s  already  substantial  cost  growth.  Affordability  pressures  could  also  still  affect 
other  large  programs  in  our  set,  particularly  JSF. 

3.  Utility  of  Reserve  Budget  Provisions 

Many  studies  of  the  problems  in  defense  acquisition  have  highlighted  the  lack  of 
an  explicit  mechanism  within  the  DoD  for  programming  and  budgeting  financial  reserves 
for  unforeseen  contingencies  For  complex  defense  system  developments,  such  as  is 
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normally  the  case  for  MDAPs,  the  ability  to  accurately  estimate  development  and 
production  costs  is  tenuous,  even  under  the  best  of  circumstances.  Faced  with  such 
inherent  and  unavoidable  uncertainties,  it  is  good  management  practice  to  maintain  a 
reserve  of  funds  that  can  be  drawn  on  to  meet  unexpected  needs.  If  no  reserves  exit  and 
the  unexpected  contingency  arises,  then  the  required  funds  might  be  taken  from  other, 
presumably  lower  priority,  programs.  That  can  be  highly  disruptive  to  those  programs, 
resulting  in  stretch-outs  of  many  programs  and  ultimately  increased  costs.  Alternatively, 
the  program  in  question  can  attempt  to  execute  without  additional  funding  by  taking  risky 
shortcuts,  such  as  excessively  concurrent  scheduling,  or  curtailment  of  systems 
engineering  and  risk  reduction  efforts.  Either  approach  will  likely  be  detrimental,  as  we 
have  seen  in  the  programs  we  examined. 

Our  study  of  the  eleven  troubled  programs  did  not  explicitly  identify  lack  of 
reserves  as  a  significant  problem.  The  problems  encountered  by  each  of  these  programs 
were,  in  most  cases,  so  large  and  not  of  a  nature  for  which  the  injection  of  any  reasonable 
amount  of  “reserve”  funds  would  have  kept  them  on  schedule.  Indeed,  most  slipped 
primarily  for  technical  reasons  -  not  for  want  of  funding.  But  they  eventually  needed 
more  funding  to  be  continued,  and  that  funding  may  well  have  come  from  reductions  in 
the  funding  of  other  healthier  programs,  but  our  research  did  not  extend  to  examining 
such  effects. 

4,  Cost  Estimates  for  Development  Contracts 

Most  new  MDAPs  involve  years  of  “invention”  activities  with  costs  that  can  only 
be  roughly  approximated  in  advance.  Furthermore,  the  cost  of  development  is  usually 
only  a  small  fraction  of  the  total  cost  for  programs  that  lead  to  years  -  or  decades  -  of 
production,  such  as  most  aircraft,  ships,  ground  vehicles  and  expendable  weapons.  Both 
of  these  facts  argue  for  a  de-emphasis  on  contractor  estimates  of  the  development  costs  in 
the  initial  selection,  in  favor  of  emphasizing  the  likely  quality  of  the  development  effort 
and  the  cost  and  performance  of  the  resulting  system  when  it  reaches  production. 

But  Congressionally-driven  Federal  procurement  laws  and  procedures  are  sharply 
biased  toward  the  comparison  of  cost  bids  -  even  for  cost-reimbursable  development 
contracts.  Cost  “shootouts”  are  perfectly  reasonable  when  the  government  is  purchasing 
known  commodities  or  fully  defined  end  items  that  require  little  or  no  engineering 
development.  But  picking  development  contractors  based  primarily  on  their  own 
estimated  development  cost  is  highly  risky.  Some  senior  industry  executives  have  gone 
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so  far  as  to  suggest  that  the  government  should  ignore  bidders’  cost  estimates  when 
making  an  award,  and  concentrate  instead  on  the  proven  quality  of  recent  work  as  well  as 
the  innovative  ideas  and  expertise  that  the  contractor  would  bring  to  the  program.^ ^ 

Indeed,  the  government  has  attempted  to  move  away  from  “cost  shoot  outs” 
towards  “best  value”  contracting  for  complex  development  programs,  but  the  frequency 
of  sustained  protests  tends  to  inhibit  this  approach.  A  black  and  white  cost  comparison  is 
certainly  simpler,  more  understandable,  and  defendable  to  the  Congress.  But  it  doesn’t 
always  get  the  best  results. 

C.  CONCLUDING  REMARK 

While  each  program  we  examined  was  afflicted  by  a  unique  cost-growth 
syndrome,  together  they  support  a  unifying  insight:  cost  growth  in  each  of  them  could 
have  been  greatly  reduced  or  eliminated  if  policies  and  procedures  previously  developed 
and  promulgated  for  that  purpose  had  been  more  rigorously  enforced.  In  most  cases, 
rigorous  enforcement  would  have  delayed  the  system’s  entry  into  (but  not  necessarily  its 
completion  of)  development.  In  several  cases  it  would  have  forced  a  system  with 
expanded  requirements  to  re-qualify  for  development.  And  it  would  have  raised  the 
standard  of  systems  definition  and  systems  engineering  to  levels  that  would  have  made 
reliable  cost  estimation  practicable  and  successful  development  more  likely.  The 
recommendations  presented  in  the  next  chapter  are  intended  to  re-instill  the  necessary 
rigor  to  the  process. 


Report  of  the  Defense  Seience  Board/Air  Force  Scientific  Advisory  Board  Joint  Task  Force  on 
Acquisition  of  National  Security  Space  Programs,  May  2003. 
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III.  RECOMMENDATIONS 


The  following  recommendations  are  intended  to  help  prevent,  or  least  alleviate, 
each  of  the  major  causes  of  cost  growth  identified  in  our  study.  They  are  directly 
supportive  of  the  goals  established  by  the  WSARA  2009.^2  These  detailed 
recommendations  flow  from,  but  are  not  tightly  tied  to,  specific  causes  identified  in  each 
of  the  case  studies. 

Several  of  our  recommendations  entail  increasing  the  government’s  expertise  in 
systems  engineering  and  management  -  a  process  requiring  more  senior,  experienced 
people.  This  appears  quite  feasible  under  the  recently  increased  authorized  size  of  the 
defense  acquisition  corps,  but  it  will  take  years  to  complete.  In  the  meantime, 
experienced  people  who  are  free  of  conflicts  of  interest  can  likely  be  “borrowed”  for 
specific  reviews  from  government  laboratories.  University  Affiliated  Research  Centers, 
and  Federally  Funded  Research  and  Development  Centers,  all  of  which  are  within  the 
purview  of  the  DAE. 

A,  IMPROVE  DEFENSE  ACQUISITION  EXECUTIVE  MANAGEMENT 
OVERSIGHT 

The  DAE  should  establish  a  greater  sense  of  commitment  and  accountability 
among  the  leaders  of  the  acquisition  team  through  a  formal  process  for  tracking 
commitments  made  at  DABs  and  at  other  OSD  reviews.  This  process  should  include 
tracking  the  implementation  of  ADM  directives  in  both  contracts  and  budget  documents. 
Tracking  should  become  a  regular  feature  of  a  revitalized  DAES  review  process  that  is 
designed  to  identify  MDAP  problems  early  enough  for  the  DAE  to  take  appropriate 
corrective  action.  A  useful  top  level  outline  of  a  tracking  process  is  shown  in  Eigure  2 
below. 
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Weapon  Systems  Aequisition  Reform  Aet  of  2009,  Publie  Law  1 1 1-23-May  22,  2009,  1 1 1th  Congress. 
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Figure  5.  Template  Used  to  Track  Record  of  Progress  for  MDAPs 

B.  STRENGTHEN  THE  ANALYTIC  BASIS  FOR  SETTING  MAJOR  DEFENSE 
ACQUISITION  PROGRAMS  REQUIREMENTS 

The  DAE  should  work  closely  with  the  Director  CAPE  to  reinvigorate  mission- 
area  type  analyses  of  the  capabilities  needed  to  meet  the  Secretary’s  defense  strategy  to 
identify  both  appropriate  changes  in  ongoing  MDAPs,  and  the  key  performance 
requirements  of  any  new  materiel  needs.  These  analyses  should  include  an  appropriate 
range  of  materiel  and  non-materiel  alternatives,  including  programs  recommended  by  the 
DoD  Components  and  by  the  Joint  Staff.  Essential  to  this  analysis  is  sufficient  technical 
expertise  to  permit  accurate  assessment  of  the  feasibility,  risks,  and  likely  cost  of  any 
candidates  proposed  for  new  materiel  solutions. 

C.  RENAME  AND  RESTART  PROGRAMS  WITH  MAJOR  SCOPE  CHANGES 

Whenever  a  major  change  is  being  considered  for  an  MDAP,  the  DAE  should 
conduct  a  DAB  review  that  includes  potentially  renaming  and  formally  restarting  the 
program  at  an  appropriate  milestone  with  a  new  Acquisition  Program  Baseline  and 
Selected  Acquisition  Report. 


58 


D,  STRENGTHEN  TECHNICAL  OVERSIGHT 


The  DAE  should  establish  a  CAIG-like  Systems  Engineering  Improvement  Group 
to  conduct  independent  reviews  of  the  proposed  design  concept  and  development  plan  of 
new  MDAPs.  The  group  should  comprise  experienced  systems  designers,  and  the  reviews 
should  be  scheduled  early  enough  to  help  the  CAIG  by  strengthening  preparation  of  the 
CARD.  One  of  its  goals  should  be  ensuring  that  the  development  funding  plan  is 
adequate  to  limit  the  risks  at  the  next  milestone  to  acceptable  levels. 

E.  IMPROVE  CONTRACTOR  SELECTION 

The  DAE  should  strengthen  the  government’s  ability  to  assess  the  realism  of 
contractor  proposals  for  MDAP  development  and  to  make  awards  based  on  “best  value.” 
To  this  end,  source  selection  for  cost-type  development  contracts  should  de-emphasize 
the  contractor’s  estimates  of  the  likely  cost  in  favor  of  an  increased  emphasis  on  the 
contractor’s  technical  approach  and  overall  qualifications  and  the  government’s  estimate 
of  the  likely  cost  for  that  contractor  to  successfully  complete  development  in  a  way  that 
leads  to  affordable  production  costs.  In  addition  the  DAE  should  strengthen  the 
qualifications  of  the  government’s  pre-award  survey  teams. 

Another  step  would  be  to  strengthen  the  process  for  ensuring  that  only  clearly 
qualified  contractors  are  solicited  for  formal  bids  by  increasing  the  use  of  the  past 
performance  database. 

F.  STRENGTHEN  AFFORDABILITY  ASSESSMENTS 

To  limit  “over-programming”  of  MDAPs  beyond  the  EYDP  years  and  the 
resulting  “bow  wave”  of  unaffordable  programs,  the  DAE  should  work  with  the  Director 
CAPE  to  ensure  that  the  DPP  is  updated  at  least  annually.  Then  it  should  be  used 
explicitly  at  each  DAB  to  assess  long-term  affordability  within  the  expected  acquisition 
portfolio  funding  totals.  The  annual  Program  Objective  Memorandum  POM  review 
should  include  an  assessment  of  the  longer-term  affordability  of  the  entire  acquisition 
program,  as  well  as  a  verification  of  the  implementation  of  the  DAB-approved  funding 
plans  for  each  MDAP. 

G.  AVOID  UNPRODUCTIVE  PROTOTYPING 

Acquisition  executives  should  ensure  that  full  funding  is  provided  for 
competitively  prototyping  of  those  elements  of  a  new  system  for  which  such  prototyping 
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can  be  shown  to  be  eost  effeetive.  Do  not  rely  on  prototyping  as  a  substitute  for  robust 
systems  engineering. 

H.  AVOID  INEFFECTIVE  OR  MISGUIDED  REFORMS 

When  eonsidering  potential  new  “reforms”  of  the  DoD  aequisition  management 
system,  the  DAE  should  first  subject  proposals  to  controlled  trials  on  one  or  a  few 
appropriate  programs.  Only  when  the  data  from  sueh  “pilot  programs”  have  demonstrated 
the  effeetiveness  of  the  intended  improvement  should  eonsideration  be  given  to  the  wider 
applieation  of  the  reform.  The  results  of  prior  reform  efforts  should  be  thoroughly 
assessed  before  failed  approaehes  are  reeonsidered. 

I.  FINAL  NOTE 

Implementation  of  these  reeommendations  will  require  no  ehanges  to  existing 
legislation,  and,  with  the  possible  exeeption  of  the  ereation  of  the  SEIG,  no  major 
organizational  changes  within  the  Pentagon.  Implementation  will,  however,  require  a 
eonsiderable  inerease  in  the  diligenee  with  whieh  the  DAE,  his  staff,  and  other  members 
of  the  defense  aequisition  team  implement  longstanding  polieies  (as  refieeted  in  extant 
DoD  direetives)  and  best  aequisition  praetiees.  Those  praetiees  are  rooted  in  the  report  of 
the  1986  Paekard  Commission,  have  been  expanded  by  numerous  subsequent  studies,  and 
are  further  reinforeed  by  the  Weapon  Systems  Aequisition  Reform  Aet  of  2009. 
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